LinuxJournal January 2016 VK Com Stopthepress

™
AUTOMATE
Full
Disk
Encryption
+
Since 1994: The Original Magazine of the Linux Community JANUARY 2016 | ISSUE 261 | www.linuxjournal.com
IMPROVE
File Transfer Enhance
Client-Side
Security Performance
for Users
Making
Sense of
Profiles and
RC Scripts
ABINIT for
Computational
Chemistry
Research
Leveraging
Ad Blocking
Audit Serial WATCH:

ISSUE
OVERVIEW
Console Access
V
LJ261-January2016.indd 1 12/17/15 8:35 PM

Practical books
for the most technical
people on the planet.
GEEK GUIDES
Download books for free with a

simple one-time registration.
http://geekguide.linuxjournal.com

Improve Finding Your
Business Way: Mapping
Processes with Your Network
an Enterprise to Improve
Job Scheduler Manageability
Author: Author:
Mike Diehl Bill Childers
Sponsor: Sponsor:
Skybot InterMapper
DIY Combating
Commerce Site Infrastructure
Author:
Sprawl
Reuven M. Lerner Author:
Sponsor: GeoTrust Bill Childers
Sponsor:
Puppet Labs
Get in the Take Control

Fast Lane of Growing
with NVMe Redis NoSQL
Author: Server Clusters
Mike Diehl Author:
Sponsor: Reuven M. Lerner
Silicon Mechanics Sponsor: IBM
& Intel
Linux in Apache Web

the Time Servers and
of Malware SSL Encryption
Author: Author:
Federico Kereki Reuven M. Lerner
Sponsor: Sponsor: GeoTrust
Bit9 + Carbon Black

CONTENTS JANUARY 2016
ISSUE 261
FEATURES
50 Secure File Transfer 72 Transferring Conserver
Use RFC 1867, thttpd and Stunnel Logs to Elasticsearch
to improve security. Auditing serial console access in
Charles Fisher real time.
Fabien Wernli
4 / JANUARY 2016 / WWW.LINUXJOURNAL.COM

COLUMNS
26 Reuven M. Lerner’s
At the Forge
Client-Side Performance
32 Dave Taylor’s
Work the Shell
Planetary Age
36 Kyle Rankin’s
Hack and /
Full Disk Encryption
40 Shawn Powers’ 17
The Open-Source Classroom
Profiles and RC Files
86 Doc Searls’ EOF

What We Can Do with
Ad Blocking’s Leverage
IN EVERY ISSUE
8 Current_Issue.tar.gz
18
10 Letters
14 UPFRONT
24 Editors’ Choice
46 New Products
ON THE COVER
0TWYV]L-PSL;YHUZMLY:LJ\YP[`W
(\KP[:LYPHS*VUZVSL(JJLZZW
(\[VTH[L-\SS+PZR,UJY`W[PVUW
,UOHUJL*SPLU[:PKL7LYMVYTHUJLMVY<ZLYZW
4HRPUN:LUZLVM7YVMPSLZHUK9*:JYPW[ZW
()050;MVY*VTW\[H[PVUHS*OLTPZ[Y`9LZLHYJOW
3L]LYHNPUN(K)SVJRPUNW
24
LINUX JOURNAL (ISSN 1075-3583) is published monthly by Belltown Media, Inc., PO Box 980985, Houston, TX 77098 USA. Subscription rate is $29.50/year. Subscriptions start with the next issue.
WWW.LINUXJOURNAL.COM / JANUARY 2016 / 5

Executive Editor Jill Franklin
jill@linuxjournal.com
Senior Editor Doc Searls
doc@linuxjournal.com
Associate Editor Shawn Powers
shawn@linuxjournal.com
Art Director Garrick Antikajian
garrick@linuxjournal.com
Products Editor James Gray
newproducts@linuxjournal.com
Editor Emeritus Don Marti
dmarti@linuxjournal.com
Technical Editor Michael Baxter
mab@cruzio.com
Senior Columnist Reuven Lerner
reuven@lerner.co.il
Security Editor Mick Bauer
mick@visi.com
Hack Editor Kyle Rankin
lj@greenfly.net
Virtual Editor Bill Childers
bill.childers@linuxjournal.com
Contributing Editors
)BRAHIM (ADDAD s 2OBERT ,OVE s :ACK "ROWN s $AVE 0HILLIPS s -ARCO &IORETTI s ,UDOVIC -ARCOTTE
0AUL "ARRY s 0AUL -C+ENNEY s $AVE 4AYLOR s $IRK %LMENDORF s *USTIN 2YAN s !DAM -ONSEN
President Carlie Fairchild

publisher@linuxjournal.com
Publisher Mark Irgang

mark@linuxjournal.com
Associate Publisher John Grogan

john@linuxjournal.com
Director of Digital Experience Katherine Druckman

webmistress@linuxjournal.com
Accountant Candy Beauchamp

acct@linuxjournal.com
Linux Journal is published by, and is a registered trade name of,

Belltown Media, Inc.
PO Box 980985, Houston, TX 77098 USA
Editorial Advisory Panel

Nick Baronian
Kalyana Krishna Chadalavada
"RIAN #ONNER s +EIR $AVIS
-ICHAEL %AGER s 6ICTOR 'REGORIO
$AVID ! ,ANE s 3TEVE -ARQUEZ
$AVE -C!LLISTER s 4HOMAS 1UINLAN
#HRIS $ 3TARK s 0ATRICK 3WARTZ
Advertising
E-MAIL: ads@linuxjournal.com
URL: www.linuxjournal.com/advertising
PHONE: +1 713-344-1956 ext. 2
Subscriptions
E-MAIL: subs@linuxjournal.com
URL: www.linuxjournal.com/subscribe
MAIL: PO Box 980985, Houston, TX 77098 USA
LINUX is a registered trademark of Linus Torvalds.

Where every interaction matters.
break down
your innovation barriers
power your business to its full potential
When you’re presented with new opportunities, you want to focus on turning
them into successes, not whether your IT solution can support them.
Peer 1 Hosting powers your business with our wholly owned FastFiber NetworkTM,
solutions that are secure, scalable, and customized for your business.
Unsurpassed performance and reliability help build your business foundation to

be rock-solid, ready for high growth, and deliver the fast user experience your
customers expect.
Want more on cloud?

Call: 844.855.6655 | go.peer1.com/linux | Vew Cloud Webinar:
Public and Private Cloud | Managed Hosting | Dedicated Hosting | Colocation

Current_Issue.tar.gz
2016: a
SHAWN POWERS
Long Year
I
know you’re expecting a sarcastic calculations this issue and explains how
comment about an election year in to determine your age on other planets
the US making it seem longer than programmatically. There’s more to it than
normal, but no, 2016 is literally a longer that, but whether you plan to stay on Earth
year than most. (Although that bit about or migrate to Mars, learning to calculate
it seeming even longer has some merit.) with the date command will be a useful
What better way to start this bonus-sized skill no matter where you live. Speaking of
year than with an issue of Linux Journal? time, Kyle Rankin gives a lesson in how he
I’m not a fan of resolutions, but I do have spent many hours saving a few minutes.
a challenge for you: learn something new More specifically, he teaches how to use
this year. Personally, I plan to learn more the Debian preseed procedure to automate
about development. I dabbled in 2015, disk encryption and partition creation. It
and it’s given me the urge to learn more. sounds like something that wouldn’t be
Reuven M. Lerner is the perfect author too complicated to automate, but Kyle
to join on a journey like that, and this found it was a messy rabbit hole. His
month, he teaches how to help improve column should at least provide a flashlight
client-side performance on your Web if you decide to delve into a similar hole.
applications. Sure, we could buy everyone I took a note from my own challenge
faster computers, but Reuven shows that this month and learned the exact way Linux
there are better (and cheaper) ways to systems deal with profile and RC files. It
accomplish client-side improvements. seems like a trivial thing to learn about, but
Dave Taylor does some really cool it turns out that the procedures for loading
profiles and such are fairly complicated. I
VIDEO: was tired of just copy/pasting information
V
Shawn Powers runs into files without knowing exactly why

through the latest issue.
some information goes into profiles and

CURRENT_ISSUE.TAR.GZ
some into RC files, so I decided to get to then goes on to describe the process for
the bottom of how those preference files consolidating log files into searchable
are loaded. This month, I share the fruit of archives and even shows how to integrate
my labor and hope to demystify the shell- console logs into a real-time monitoring
based config files for everyone reading. solution. If you manage a large number of
Encrypting filesystems and salting servers via console or serial (even over the
hashes are common ways to protect LAN), you’ll want to read his article.
DATA ON A SERVER 1UITE HONESTLY WERE Doc Searls finishes the issue by discussing
beginning to see the value in encrypting the ramifications of ad blocking on the
local data, and it’s becoming common modern Internet. If you browse the Web,
for servers to be secured more than ever chances are pretty good that you use an
before. Unfortunately, most security ad blocker to make your experience more
breaches aren’t happening on the local pleasant. Blocking ads means blocking
machines; rather, they’re happening over revenue for content creators, and rather
the network. It doesn’t matter how secure than pretending it’s not an issue, we need
your local filesystem might be, if you’re not to figure out how to respond in a way that
transmitting and receiving data in a secure is useful both to consumers and content
way, no amount of local encryption will creators. As usual, Doc has incredible
protect your data. Charles Fisher not only insight, and you’ll want to check it out.
exposes the weaknesses with traditional file This first issue of Linux Journal in 2016
transfer methods, but he also explains how may be brand new, but it still has all the
to shore up network transfers when sending tech tips, product reviews and helpful
and receiving data. Whether you consider information you’ve come to expect month
your data sensitive or not, there’s no after month. Whether the new year means
reason to adopt insecure methods in your ice and snow or sunshine and roses in your
environment. Charles shows how to make part of the world, we hope this issue helps
sure you keep your private data private, start it off on a good note. We’ll see you
even when you send it across the Internet. again next month, when February grows an
Fabien Wernli also discusses security this extra day and is almost a full-size month!Q
month, but rather than securing network
transfers, he covers how to manage log Shawn Powers is the Associate Editor for Linux Journal .
files for console connections. Keeping track He’s also the Gadget Guy for LinuxJournal.com, and he has
of serial connections to the server console an interesting collection of vintage Garfield coffee mugs.
can be challenging when your server Don’t let his silly hairdo fool you, he’s a pretty ordinary guy
number increases, but thanks to syslog-ng, and can be reached via e-mail at shawn@linuxjournal.com.
you’re able to log that information. Fabien Or, swing by the #linuxjournal IRC channel on Freenode.net.

letters
ALL )0S INTO A HASHIP WHICH QUICKLY
becomes too large. The script is at
https://github.com/zuikway/tlj_blocklist.
—Wayne Shumaker
Server Hardening, II
Greg Bledsoe missed one small thing
that can increase a server’s security:
reduce the amount of network traffic
a server must process:
iptables -t mangle -I PREROUTING -m state --state INVALID -j DROP
INVALID packets are those that must

Server Hardening—ipset:set belong to an established connection,
Regarding Greg Bledsoe’s “Server yet netfilter has no connection
Hardening” article in the November recorded for it. They are “spurious”
2015 issue: I created a modified script packets that cannot be delivered, so
for generating ipset blocklists. Namely they should be dropped as early as
it creates a set of ipsets, one a hash:net possible. It isn’t worth spending one
and the other a hash:ip. The script extra CPU cycle on these packets.
generates a second script called blset.sh, Although it won’t eliminate the
which adds the IP addresses to the ipset ill effects of a DDoS attack, it can
hashes. The blset.sh script first adds all significantly reduce the time the CPU
the hash:net entries from the various spends handling INVALID packets.
sources, then the hash:ip set is created, —Neal
but entries are not added if they already
exist in the hash:net set. Find Words
Dave Taylor’s Work the Shell column
The new script does not exceed the in the September–November 2015
ipset size limit. The suggested script issues covers a fun toy program near
in Greg’s Linux Journal article puts and dear to my heart. I’ve been using

[ LETTERS ]
a very similar de-jumbling algorithm word lists I found on-line). In my

to strengthen my scripting in Perl language du jour, I construct a massive
and Python—although I must admit hash keyed on the alphabetized words,
I haven’t been ambitious enough to with an array of matching original
implement it in bash! It was cool to see words as the value. For example:
Dave use the nearly the same approach
I came up with myself. I figured it $list{'abt'} -> ['bat', 'tab']
might be interesting to share my own
variation to the same problem. All in all, this approach takes only a few
seconds on a five-year-old laptop, and
Considering that modern machines 21MB of RAM for the data structure.
are overkill for most scripts, I started
off simply alphabetizing the entire The next fun part was digging into
dictionary (first in /usr/share/dict/words, my computer science background
and later a set of professional Scrabble and using a recursive algorithm to
Linux Journal
Archive 1994–2015
NOW AVAILABLE!
www.linuxjournal.com/archive

[ LETTERS ]
deconstruct the input letter sets by versions, forcing me to stick to a certain

calling the same function minus a version or adapt my awk|perl|grep
different letter each time and looking processing of the text output. Text
up the result in the hash. Putting the output for the Python CLI was bigger
input function into a loop (checking and a bit trickier to parse well—enter
FOR %/& OR hQv FOR TERMINATION ALLOWS JSON output. As Kyle writes, the Python
you to perform multiple searches CLI offers the option of different
against the hash you’ve spent several outputs, including JSON. It’s a slightly
busy CPU-seconds constructing. steeper learning curve, but using the
*3/. OUTPUT TOGETHER WITH THE JQ *3/.
Keep on hacking! command-line parser makes processing
—Chandler Wilkerson anything from the CLI straightforward
and keeps me safe from EC2 CLI adding
Dave Taylor replies: Great to hear fields or new lines, etc., that may break
from you, Chandler, and glad my column by text processing! One can always
brought you some enjoyment as you script things prettier, but being a
realized we’d taken the same algorithmic one-liner fan, one can, for example, get
approach to the word jumble algorithm! all the volume IDs for one’s servers:
AWS EC2 VPC CLI aws ec2 describe-instances | jq -r
Thanks for an excellent journal. I ´'.Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId'
really enjoy it and love the digital

version on my Kindle. Taking it a little further, snapshot every
EBS volume, but only if it does not
The reason I’m writing is just a general belong to a certain tag (or do it the
hint to Kyle Rankin’s great article on other way around and snapshot only a
the EC2 CLI in the October 2015 issue. given tag) and snapshot only those that
I have myself gone through an identical are mounted on a given device name:
process for exactly the same reasons in
changing to the Python CLI. The only aws ec2 describe-instances | jq -r '.Reservations[].Instances[] |
thing I chose to do differently in the ´select(contains({Tags: [{Key: "SomeKey",Value:
end was processing the output. I, on ´"SomeValue"} ]}) | not) | .BlockDeviceMappings[] |
occasion, had issues in processing the ´select(contains({DeviceName: "/dev/sda"})) | .Ebs.VolumeId'
text output of the Java CLI in that it ´| parallel aws ec2 create-snapshot
sometimes changed slightly between ´--description "backup_`date +\%Y\%m\%d`" --volume-id

[ LETTERS ]
parallel is a great trick to call the command on

every volume ID. I would often use xargs and give
multiple IDs in one call, but with the Python CLI, I
At Your Service
could give each call only one volume ID. I add the SUBSCRIPTIONS: Linux Journal is available
in a variety of digital formats, including PDF,
date to the description for a better overview of .epub, .mobi and an on-line digital edition,
as well as apps for iOS and Android devices.
snapshots and a simple way to monitor and delete Renewing your subscription, changing your
e-mail address for issue delivery, paying your
given snapshots. invoice, viewing your account details or other
subscription inquiries can be done instantly
on-line: http://www.linuxjournal.com/subs.
Then, I would also have a similar simple one-liner E-mail us at subs@linuxjournal.com or reach
us via postal mail at Linux Journal, PO Box
to clean up old snapshots and monitor that all 980985, Houston, TX 77098 USA. Please
remember to include your complete name
snapshots are successful. and address when contacting us.
ACCESSING THE DIGITAL ARCHIVE:
Keep up the good work! Your monthly download notifications

will have links to the various formats
—Elfar and to the digital archive. To access the
digital archive at any time, log in at
http://www.linuxjournal.com/digital.
Photo of the Month LETTERS TO THE EDITOR: We welcome your

letters and encourage you to submit them
Mateo from at http://www.linuxjournal.com/contact or
Argentina, already mail them to Linux Journal, PO Box 980985,

Houston, TX 77098 USA. Letters may be
supporting Linux the edited for space and clarity.
first day of his life. WRITING FOR US: We always are looking
for contributed articles, tutorials and
—Gaston real-world stories for the magazine.
An author’s guide, a list of topics and
due dates can be found on-line:
http://www.linuxjournal.com/author.
FREE e-NEWSLETTERS: Linux Journal

editors publish newsletters on both
a weekly and monthly basis. Receive
late-breaking news, technical tips and
tricks, an inside look at upcoming issues
and links to in-depth stories featured on
PHOTO OF THE MONTH http://www.linuxjournal.com. Subscribe
for free today: http://www.linuxjournal.com/
Remember, send your Linux-related photos to enewsletters.
ljeditor@linuxjournal.com! ADVERTISING: Linux Journal is a great

resource for readers and advertisers alike.
Request a media kit, view our current
editorial calendar and advertising due dates,
or learn more about other advertising
WRITE LJ A LETTER and marketing opportunities by visiting

us on-line: http://ww.linuxjournal.com/
We love hearing from our readers. Please advertising. Contact us directly for further
information: ads@linuxjournal.com or
send us your comments and feedback via +1 713-344-1956 ext. 2.
http://www.linuxjournal.com/contact.

UPFRONT NEWS + FUN
diff -u
What’s New in Kernel Development
There’s an ongoing impulse among heavy, inefficient battery technology,
a diversity of developers to be able there’s a big incentive to figure out
to compile some or all of the Linux ways to save power. One possibility
kernel as a library, so that a piece of is to turn off portions of hardware
software could use kernel services when they’re currently not in use,
and APIs while running under a like a phone’s touchscreen when the
different kernel entirely, or a different phone is in your pocket.
operating system. The difficulty lies in knowing exactly
This time, the impulse came which piece of hardware to turn off,
from Octavian Purdila, creator of and when. If there’s a clear user action,
the Linux Kernel Library (LKL), like flipping closed a flip-phone, the
essentially an entire kernel compiled problem is simplified. Irina Tirdea
as a static library. He distinguished LKL recently tried to recognize such actions
from projects like User Mode Linux and come up with mechanisms to
(UML), saying that LKL was more respond to them properly. She posted
lightweight, having no infrastructure some patches to do this.
REQUIREMENTS OR NEEDING ANY PARTICULAR Octavian Purdila, also working
sort of runtime environment. on the project with Irina, described
A bunch of folks expressed interest, a target scenario as being when a
especially in terms of interacting touchscreen has been blanked but is
with similar projects like libOS still aware of the user’s touch—through
and libguestFS. And, Richard the fabric of a pocket, for example.
Weinberger remarked that LKL The goal of the patches, he said,
seemed to solve UML’s biggest pain would be to save power by turning off
points: the need to use ptrace() to all the hardware associated with that
handle system calls and to do virtual screen, and turn everything on again
memory management using SIGSEGV. when the user activates the device.
In a device-centric world with The problem with this sort of feature

[ UPFRONT ]
is that it could be implemented along Meanwhile, Lukasz Pawelczyk was

any of a number of different layers working on code specifically to support
of the kernel code. The ideal location that same security information.
could make the difference between a A debate sprang up over the
complex, easily broken implementation particular context involved. Andy
and a simple, efficient implementation. Lutomirski suggested that if a
Several folks felt that Irina and filesystem contained a user’s own data,
Octavian’s approach was in the wrong it would be fine to override security
part of the kernel, and the discussion features, on the grounds that users
devolved into a consideration of should be able to do what they wanted
completely different approaches. with their own data. While Casey
No consensus arose, although Schaufler replied that the kernel
the allure of power-savings will shouldn’t care what the user knew
undoubtedly keep the debate alive. about the data, it had to follow the
Mounting a filesystem under a security protocols or else it wouldn’t be
virtual machine can be tricky. Security able to enforce them at all.
privileges and restrictions need to On the other hand, as Eric pointed
be respected, or else a filesystem out, filesystems like FAT and Minix
could become a vector of attack weren’t capable of storing the same
by a malicious user. This particular type of security information as more
area of kernel development also modern filesystems. There had to be a
tends to have a wide appeal among way, he said, to mount such filesystems
companies trying to support their WITHOUT REQUIRING THEM TO SUPPORT
products, so it’s possible for a variety security features they couldn’t support.
of developers to find themselves It’s an ongoing debate. Security
working at cross purposes and need trumps all other considerations,
to accommodate each other before including dire need, so an issue
their patches can be accepted. like unprivileged filesystem mounts
Seth Forshee and Eric Biederman, inevitably will involve a consideration
for example, recently wrote some of the specific context in which a user
patches to allow mounting Ext4 and might try to do something. Often
FUSE filesystems by unprivileged users, there’s some kind of crazy nuance
ignoring the security information that that makes something feasible when
otherwise might prevent those users you could have sworn it never would
from accessing that data. be feasible. —ZACK BROWN

[ UPFRONT ]
Non-Linux FOSS:
Open-Source Windows?
I have mixed emotions
about ReactOS. It’s
open source. It’s freely
available. But, its goal is
to be binary-compatible
with Windows! ReactOS
is not a Linux operating
system. In fact, it
doesn’t share the UNIX
architecture at all. It
looks like Windows NT,
and it behaves much like
Windows NT.
It’s just odd!
The best way I can
think to describe it is to imagine if code. I’m personally not convinced
Wine evolved into an entire operating that ReactOS is a better idea than
system that booted on hardware Wine running inside Linux, but I’m sure
instead of running inside Linux. running it as its own operating system
That’s basically what ReactOS feels will provide possibilities that just can’t
like. It’s not ready for prime time happen in a Wine environment. The
(and the developers make that very folks at ReactOS provide installers
clear—it’s alpha software), but it’s AND PREBUILT 6- INSTANCES THAT CAN BE
worth checking out. Since it’s early in launched in order to try it out on your
the development process, if you get existing system. Whether you are just
involved now, you can have a say in morbidly curious about a non-Windows
what compatibilities get priority. Windows or are interested in getting
ReactOS is the perfect solution for involved in the development, go to
folks who need to run Windows apps, http://reactos.org for more details.
but absolutely refuse to run Microsoft —SHAWN POWERS

[ UPFRONT ]
Android Candy:
Quality Time, or Not?
This is the season of resolutions,
and in the technological world we
live in, spending time off-line is a
difficult but healthy activity. The
problem is our lives have become so
intertwined with our phones that
it’s easy to whip out our cell phones
inadvertently to check our social
NETWORKS QUICKLY
4HE 1UALITY4 IME APP IS DESIGNED
to help curb the habit just a bit.
Ironically, it’s an Android app
designed to help you stop using
Android apps so much. Still, it’s
just geeky enough to make limiting (Photo from http://qualitytimeapp.com)
technology time a fun endeavor.
If you like graphs, data, numbers TO GIVE 1UALITY4 IME A TRY )F YOU
AND GOALS 1UALITY4 IME CAN HELP just want to see how much time
you identify where you spend most you spend on various applications
of your time on-line and then on your Android device, you
assist in lessening your face time SHOULD TRY 1UALITY4 IME AS WELL )
with FaceT ime (okay, not actually found the data alone worth the
FaceT ime, since that’s an Apple installation, and it inspired me to
app, but the word play was too spend a little less time texting my
fun to leave out). kids and a little more time talking
If you’re forgetting what your to them (while they text their
family members actually look friends—baby steps...).
like, or if you’re surprised to see Check it out at
your friends as anything but their http://qualitytimeapp.com.
on-line avatars, you really need —SHAWN POWERS

[ UPFRONT ]
Dear Kodi, Where’s

My Surround?!?!
I love Kodi. (This is just an evolution we found its interface to be

of my love for XBMC, since it’s the cumbersome and the transcoding
same thing with a new name.) In for local media frustrating.
fact, although I’ve expressed my So during the holidays, I once
love for Plex over and over (and again installed Kodi on Raspberry Pi
over) the past few years, I still use devices around my house. Using
Kodi as my main interface for the OpenELEC, the installation process
televisions in my house. We gave itself is painless. Heck, even
Plex a try as our main media center centralizing the library database was
software when it was released for painless. The frustrating part was
4 I6O BUT AFTER SEVERAL MONTHS getting 5.1 surround sound to work.

[ UPFRONT ]
On the bedroom televisions, surround sound

They
is a moot point, because I just use whatever Said It
STEREO SPEAKERS ARE INCLUDED IN THE 46 &OR OUR
main media center, however, I have a fancy Don’t watch
Sonos PLAYBAR with subwoofer and rear the clock; do
channel speakers. The only audio connection the what it does.
PLAYBAR accepts is optical audio, so I bought an Keep going.
—Sam Levenson
inexpensive HDMI audio extractor. (This one works
great: http://smile.amazon.com/dp/B00BIQER0E.)
What you
The problem is that when Kodi is set to 5.1
do today
audio output, the center channel is missing!
can improve
There’s a bit of disagreement as to whether it’s a all your
bug in Kodi/OpenELEC or just a result of optical tomorrows.
audio supporting only two channels of audio. —Ralph Marston
(If that seems odd to you, it was to me too. But
apparently, it supports only two channels, which Life is 10%
contain all the surround information, or something what happens
like that.) The non-intuitive solution is to force to you and
Kodi to 2.0 audio. Although it doesn’t seem to 90% how you
make sense, I can vouch for it working. Kodi sends react to it.
—Charles R.
the audio as 2.0 stereo, which is transferred over Swindoll
optical (or HDMI, whatever you’re using), and then
the receiver decodes the surround information It does not
from that two-channel signal. matter how
The tl;dr version is that Kodi will send the slowly you go
surround sound information over two-channel as long as you
audio, so if you are missing your center channel, do not stop.
try switching to 2.0 audio. —SHAWN POWERS —Confucius
Keep your eyes

on the stars,
and your feet
on the ground.
—Theodore
Roosevelt

[ UPFRONT ]
ABINIT for Chemists

The single largest group of users install it with:
on high-performance computing
clusters has to be the chemists. Their sudo apt-get install abinit abinit-data abinit-doc
CPU-year count is definitely at the

very top of the list. Because of this The only issue with that method
heavy use, several different packages is you probably will get an older
have become standard tools that most version of ABINIT. At the time of this
computational chemistry researchers writing, the Ubuntu package installs
use. So in this article, I take an version 7.8.2, while on the Web site,
introductory look at one called you can download version 7.10.5.
ABINIT (http://www.abinit.org). If you need the latest available
ABINIT calculates the energy and code, you always can get the
structure of groups of nuclei and source code from the main home
electrons. The method used to make page and compile it yourself on your
these calculations is Density Functional local machine. In order to build it
Theory (DFT, https://en.wikipedia.org/ yourself, you need the usual utilities
wiki/Density_functional_theory). If to build other packages, such as
you want to know more about the make, libtool and autoconf. Because
underlying theory, feel free to go talk the majority of the code is written in
to your nearest computational chemist. FORTRAN, you also need a compiler
Although my exposure has been capable of compiling F90 code.
with people running ABINIT on scores This will allow you to build a basic
of machines in parallel, at least in version of ABINIT. You can include
a learning environment or for small extra functionality, such as MPI or
systems, nothing is stopping you NetCDF, if you have them available
from running it on your own desktop. on your system.
The first step, of course, is to install The main executable to run these
it on your machine. You may have calculations is called abinit . It
packages within your distribution to takes a number of input files in
make installation easier. For example, order to do the actual calculation.
on Debian-based systems, you can One of these input files is actually

[ UPFRONT ]
a file of files. It is a file that Q my.psp — the pseudopotential

contains a list of other input files used for this run.
that abinit needs to read in. The
usual filename ending is “.files”. The root names “abi”, “abo” and
If you have this input file, you can “tmp” are used to create the multiple
run your simulation with: files for each of those sections.
There are a few rules around
abinit < my_input.files >& log the input files that may cause
problems if you don’t follow
This tells abinit to read the input them. The first is that you can’t have
data from standard input (attached to tab characters in your input file. So,
the file my_input.files) and to write its be sure that your editor uses space
results to standard output (attached to characters when you press the tab
the file log). The log file only captures key. The second rule has to do with
output that gets written out to the using negative numbers. There can’t
standard output stream. There is a be any spaces between the negative
lot more output that is written out. sign and the first digit of the number.
These other output files are defined in The last formatting rule is that no
the my_input.files file. The following line can be more than 132 characters.
list is a more-detailed description of If any lines end up longer than that,
the contents: ABINIT simply will ignore the extra
content. If you get errors when trying
Q ab_in — main input file. to run your own jobs, those are the
first few places you should check.
Q ab_out — main output file. There are a massive number of
input variables that allow you to
Q abi — root filename for other control parameters around file
input files. handling, geometry, structure
optimization and response
Q abo — root filename for other functions, among many others.
output files. These input variables can be in
any order. The entire file gets
Q tmp — root filename for parsed before the calculations
temporary files. start. When you start creating

[ UPFRONT ]
your own input files, you probably access to all of the source code
will want to be able to check them and can investigate exactly how
somehow. Luckily, you can use the calculations are being done.
ABINIT itself to do this. The abinit When doing fundamental scientific
executable includes an option ( -d research, that can be very important.
or --dry-run ) to take your input You may be trying to do calculations
files and validate them without in a region where the available
starting the calculations. This algorithm is no longer valid. All of
allows you at least to catch major these calculations make assumptions
typos before wasting the time to try to simplify the calculations so
involved in doing a partial run and that they are actually doable, and
having it fail. it is very important to keep that in
Along with your own input files, mind. But, with access to the code,
describing the geometry and other you have the opportunity to make
descriptive variables, ABINIT needs changes to those algorithms to fit
input files that describe something the assumptions better that are valid
called the pseudopotential for for your problem. This open-source
your system. There are different code gives you the ability to build
types, such as Troullier-Martins on all of the past work and push
or Hartwigsen-Goedecker-Hutter it into new areas of research. Just
pseudopotentials, that can be used remember to pass these extensions
for different situations. Luckily, ABINIT and improvements on to the next
includes pseudopotentials for the group of researchers to keep pushing
entire periodic table. This means you our understanding forward.
simply can build up your molecule Interpreting the output from
by including the pseudopotentials ABINIT can be a bit of a job. There
for each of the different types of is a lot of output describing how
atoms in your system. Although it the calculated values progressed
isn’t necessary in most cases, you UNTIL THEY REACHED THE REQUESTED
can create your own for some very accuracy to the actual answer. For
specialized system if needed. example, if you are calculating
The other thing to be aware of the energy for a molecular
is that ABINIT is released under a configuration, you probably are
GPL license. This means you have interested in when the energy is

[ UPFRONT ]
at its lowest value. This will be the This is just a very basic
most stable configuration for these introduction to what is involved
nuclei and electrons. But, how do when using ABINIT. Hopefully, you
you interpret this output? Several now feel a bit more comfortable
tools are available to take the digging in to the massive
geometric portion of this output documentation and using ABINIT
and plot it so that you can see what to solve whatever molecular
the configuration actually looks like. problem you have. When you are
There also will be output describing ready, you can move on to much
how strong the various connections larger problems by using the
are between the nuclei, which you MPI capabilities in ABINIT to use
can use to see how reactive your as many machines as you have
molecule may be. available. —JOEY BERNARD
LINUX JOURNAL
on your Android device
Download the app

now from the
Google Play Store.
www.linuxjournal.com/android
For more information about advertising opportunities within Linux Journal iPhone, iPad and
Android apps, contact John Grogan at +1-713-344-1956 x2 or ads@linuxjournal.com.

[ EDITORS' CHOICE ]
Help Me, EDITORS’

CHOICE
Uncle Shawn ★
If you’re anything like me, the (Xubuntu is
holiday season is spent fixing W i-Fi usually my choice). The problem
and removing spyware. Occasionally, with helping friends and relatives
I get to install Linux for a relative with their computers over the
who is ready to give up W indows holidays is that you become their
or needs something that will run first call when something goes
on a circa-W indows 2000 computer wrong. You either can fight it

or make it easier on yourself by 6IEWER GETS THIS MONTHS %DITORS
preparing in advance. Choice award. It’s not new
) LOVE 4EAM 6IEWER )TS NOT AN software, but after a stretch of
open-source program, but it’s holidays, I’m reminded just how
free for personal use with no nice it is to have installed on all
frustrating limitations. Plus, it my relatives’ computers. Be sure
runs on W indows, OS X and Linux. to install the client before you
The best part is how easy it is to leave their houses, or else be
use. I generally don’t set up the prepared to explain software
“automatic availability” feature installation over the phone! Get
that logs the computer in to the your copy at http://teamviewer.com.
4EAM 6IEWER NETWORK AUTOMATICALLY —SHAWN POWERS
on boot. I like to use the standard
STARTUP WHICH REQUIRES USERS TO CALL
me with the code on their screen.
The best thing about Team
6IEWER IS HOW EASILY IT HANDLES LINUX JOURNAL
NAT situations. Since the software on your e-Reader
CONNECTS TO THE 4EAM 6IEWER
servers, those servers act like a
connection broker, meaning there
are no router ports to forward and
no proxies to set up. As long as the
computer is on-line, you should be
able to take over and help someone.
Again, you might not like the
ease with which you’ll be able to e-Reader
editions
help, but having access to a user’s FREE for
Subscribers
computer in real time is so much
nicer than explaining to Uncle Harry
what “right click” means. Customized Kindle and Nook
Due to its free license for personal editions now available
use, cross-platform compatibility
and incredible ease of use, Team LEARN MORE

COLUMNS
AT THE FORGE
Client-Side REUVEN M.
Performance
LERNER
Give your users a better experience by improving

client-side performance.
In my last few columns, I’ve Angular.js and React.js, assume that

covered different ways to understand, you’ll be writing your application in
analyze and improve the performance JavaScript and provide you with the
of your Web applications. I’ve objects and infrastructure necessary
shown that between your network for doing so.
connections, server hardware, If you’re worried about the
database design and HTTP server performance of your Web
configuration, you can change and application, you need to concern
improve the performance of your yourself not only with what happens
Web application—well, sort of. Web on the server, but also with what
applications, when they first started, happens in the browser. Some
were dynamic only on the server commercial performance-monitoring
side. Sure, they output HTML— solutions already take this into
and later, CSS and JavaScript—but account, allowing you to see how
the overwhelming majority of the long it takes for elements to render,
processing and computation took and then to execute, on your users’
place on the server. browsers. However, there is also
This model, of course, has changed no shortage of open-source tools
dramatically in the last decade, to available for you to check and
such a degree that you now accurately improve the ways in which your
can claim to be a Web developer and client-side programs are executing.
work almost exclusively in HTML, This month, I’m concluding this
CSS and JavaScript, with little or exploration of Web application
no server-side component. Entire performance with a survey of things
-6# FRAMEWORKS SUCH AS %MBERJS to keep in mind, as well as tools that

COLUMNS
AT THE FORGE
help ensure that you’re actually doing HTML page. That is, instead of having
what you should be. your <script> tags, whether local or
remote, at the top of your page, you
Client-Side Considerations should put them at the bottom—
Client-side code is written in JavaScript. unless it’s vital to do otherwise.
The code, whether inline in <script> Even better, you should consolidate
tags or retrieved from a remote server, your JavaScript files into a single file.
executes whenever the browser’s parser This has a number of advantages. It
gets to that part of the page. If you means the user’s browser needs to
have JavaScript at the top of the page, download a single file, rather than
it’ll be executed when the parser gets many of them. If you include all of
to it, potentially delaying the rendering the JavaScript needed on your site in
of the rest of your page. By contrast, a single file, it also means that the
if your JavaScript is at the bottom, the file needs to be loaded only a single
parser will execute it only after parsing TIME /N EVERY SUBSEQUENT PAGE LOAD
and rendering the rest of the page. This the JavaScript will be mentioned, but
is why so many developers learned to it won’t be downloaded, because it’ll
put their JavaScript commands inside a already be cached in the browser’s
“document-ready” callback function; in memory. You can make things even
that way, the code was executed only better, of course, by compressing
once the entire page had been loaded. that single JavaScript file. This
Because so many modern Web turns out to be extremely effective,
applications take place in JavaScript, because compression algorithms
the fact that you’re often loading work well with text, and especially
JavaScript from remote servers means with text that repeats itself, as
that the time it takes to render a happens with program code.
page depends not just on the server Better yet, you can run JavaScript
speed, the network bandwidth and code through a minimizer (or
the page’s complexity, but also on the “minifier”), which removes comments,
servers and networks serving such extraneous whitespace and anything
JavaScript, as well as those pages’ else that isn’t necessary for client-
complexity. As a result, it’s generally side programs to run. By minifying
considered to be good practice to JavaScript files, combining the files
load as many libraries as possible late and then compressing the resulting
in the game, at the bottom of your combination, you can dramatically

COLUMNS
AT THE FORGE
reduce the size of the JavaScript being harder to analyze. On the one hand,
sent to the user’s browser and ensure you (the developer) can download
that it is loaded only once per visit to the program, test it and check the
your Web site. performance—and then, you also can
UglifyJS, for example, can be use in-browser debugging tools to test
installed via npm : and improve things.
One of the most important tools
npm install uglify-js -g offered by both Chrome and Firefox is
the display of files being sent to the
You can run it on a file with: browser. Even if your site appears to
BE LOADING AND RENDERING QUICKLY A
uglifyjs FILENAME QUICK LOOK AT THE DOWNLOAD TIMELINE
almost certainly will be somewhere
Although because that sends output between surprising and shocking to
to stdout, you’ll likely want to redirect you. You’ll see how long it takes for
it to a file: each of the JavaScript (and CSS, and
image) files to download and, thus,
uglifyjs FILENAME > ugFILENAME.js how much time it takes between
THE USER REQUESTING YOUR PAGE AND
I took the JavaScript from my the content actually appearing on it.
PhD dissertation software and ran it This is a great way for you to identify
through both uglifyjs and gzip. The potential bottlenecks and then reduce
original 36KB file was 8.5KB after their effect on the slowness (or
compression, but 6.0KB after uglifying apparent slowness) of your site.
and compression. Although you might Even New Relic, which normally
scoff at the small size of a 36KB file is considered a (commercial) server-
in the modern world, the fact is that side performance monitor, now
each file takes time, for both the offers some client-side performance
browser and the server. The faster you checking. You place a small piece of
can get it off your server and into the JavaScript on your site; New Relic
browser, the better. collects this information, and then
tells you how long it took for your
Download Time content to get to the user’s browser
Once the JavaScript is in the user’s and how long it took to render. This
browser, things are both easier and provides a surprisingly insightful view

COLUMNS
AT THE FORGE
of whether you need to work on A FRUSTRATING AND QUIRKY LANGUAGE

improving the speed with which your the fact is that modern JavaScript
files are delivered or to optimize the IMPLEMENTATIONS RUN VERY QUICKLY
code further, such that it runs faster. assuming that you use the language in
There definitely are other options, the right way. However, it’s sometimes
but I’ve found that even the free (not hard to know where your program is
open-source, but free of charge) New spending most of its time. Fortunately,
Relic client-side benchmarking to be the Chrome developer tools (a part
QUITE USEFUL AND HELPFUL of the Chrome browser) include a
Once you have combined and profiling tool. Go to the “profile”
compressed your JavaScript files, you tab in the developer tools, and select
seriously should consider putting the CPU option before visiting your
them, as well as any other static site. You can run through your site
assets (such as CSS files and images), for a few seconds or minutes before
on a content distribution network stopping the profiling from taking
(CDN). A CDN handles only static place. Once you’ve done that, you’ll
content, but given how many large, get a (very long) indication of which
slow-to-download files are static, JavaScript programs were running,
that often can provide a significant where they came from and how much
improvement. CDNs not only have a time the CPU spent in each one.
great deal of bandwidth, but they also You similarly can ask the profiler
copy your content to multiple servers, to examine memory usage. The
using the geographically closest one more complex the application you’re
to serve content to your user. Thus, a writing, the more necessary these
user in Tokyo will receive data from a tools will be. At the same time, you’ll
local CDN server, whereas a Chicago- likely find that when you profile your
based user will receive it from a *AVA3CRIPT CODE THE MOST FREQUENTLY
different CDN server. So, using a CDN used code probably will be code you
reduces the load on your main Web didn’t write yourself, but rather that
server, while decreasing the actual and is part of the underlying framework
perceived download times. you’re using.
In Firebug, the Firefox-based
Benchmarking JavaScript debugger, you can profile a page
Although JavaScript has a (well by going to the “console” tab and
deserved, I think) reputation for being clicking on “profile”. You’ll see a

COLUMNS
AT THE FORGE
table showing how much time was perhaps necessary. It also suggested
spent in each function, and what which images could be compressed
percentage of the total time was and how much space we would save
spent there. If you’re a Chrome in so doing.
user, you can open up the developer
tools and click on the “profiles” tab. Summary
You’ll then need to choose whether Although server-side programming
you want to check CPU performance still is a vital part of the Web, the
or memory performance (in two client is where much of the action is,
different flavors). After starting and and where the user often perceives
stopping the profiler, you can analyze lags and slowness. As a result,
the resources that JavaScript used— it’s worth investing time to check
and then, of course, change your your client-side performance and
code appropriately. to address problems before your
One tool I have begun to use more users start to complain (or leave
FREQUENTLY IS 0AGE3PEED FROM 'OOGLE you without complaining). Using
This collection of tools would appear a variety of tools to check your
to be an SaaS, an updated version of performance, as well as to reduce
YSlow, which was my go-to tool for the size and time of JavaScript
many years. For example, Google’s and CSS downloads, will go a long
tools will tell you how mobile-friendly way toward improving your users’
your site is. satisfaction with your site. Q
Moreover, the PageSpeed results
always point to documentation that Reuven M. Lerner trains companies around the world in Python,
describes, in great detail, why issues PostgreSQL, Git and Ruby. His ebook, “Practice Makes Python”,
are problematic and what steps contains 50 of his favorite exercises to sharpen your Python skills.
you can take in order to fix them. Reuven blogs regularly at http://blog.lerner.co.il and tweets as
This documentation is surprisingly @reuvenmlerner. Reuven has a PhD in Learning Sciences from
well written, and it points to very Northwestern University, and he lives in Modi’in, Israel, with his
practical, clear suggestions for how wife and three children.
to improve the performance of your
JavaScript and CSS. After running
PageSpeed against one of my client’s Send comments or feedback via
sites, I found that we still had some http://www.linuxjournal.com/contact
blocking JavaScript higher up than is or to ljeditor@linuxjournal.com.

The Fourteenth Annual
Southern California Linux Expo
The Southern California Linux Expo has

grown in size and scope since it began,
and given this trend we will be in a new
venue as of 2016.
We're happy to announce the dates and

location for SCALE 14x...
January 21-24, 2016
Pasadena, CA
Featured Speakers:
Jono Bacon
Cory Doctorow
Bryan Lunduke
Mark Shuttleworth
We are pleased to be hosting the return

of the UbuCon Summit (formerly UDS)!
http://www.socallinuxexpo.org
Use Promo Code LJAD for a 30%
discount on admission to SCALE
COLUMNS
WORK THE SHELL
Planetary Age DAVE TAYLOR
For his 120th column, Dave looks at time around

the universe—programmatically, that is!
This marks my 120th column TECHNIQUES AND HOW DIVIDE AND

for Linux Journal. 120 times CONQUER CAN HELP YOU SOLVE EVEN THE
I’ve delved into the retro world of most thorny of challenges. I’m sure
shell script programming. You’ve Harry would approve!
gotten ten years of my puns and 4HIS LEAVES ME WITH THE QUESTION
wry asides—all available on the what should I do with the next ten
http://www.linuxjournal.com site YEARS 4HAT QUESTIONS ONLY SLIGHTLY
for your reading pleasure! intimidating, of course, but given that
At approximately 1,200 words/column UNIX is 45 years old (I’m using 1970
that represents 144,000 words total. as the first launch date, back on an
By comparison, Harry Potter and the old PDP-7), and Linux is 24 years old
Philosopher’s Stone is 76,944 words. (using 1991 as the date Torvalds begin
The Hobbit is 95,356 words, and Pride developing his alternative to UNIX),
and Prejudice is 122,685. So there you there’s still some time to go.
have it. In those 144k words, I could For this article, I thought it’d be fun
have created a magical universe with to talk about space. Specifically, to
an endearing young hero, or a different write a script that would tell you what
magical world with a plucky Hobbit your age would be if you lived on
adventurer (albeit reluctant adventurer), one of the other planets in our solar
or I could have written about five system. So let’s jump in!
daughters of marrying age, the charming
Mr. Bingley and the haughty Mr. Darcy. Calculating Universe Age
I have, of course, done none of Different planets revolve around the
those things. But on the bright side, sun at different rates. Earth is easy. It
I’m hoping you’ve been entertained takes the planet just a bit more than
while learning about shell script 365 days (a day being a full day/night
programming, programming cycle and based on the rotation of the

COLUMNS
WORK THE SHELL
planet) to circle the sun—an Earth year. neptune=60198.5006

But what about the other planets? pluto=90474.902
We could calculate day-length and
then number of days for each planet’s It sure takes a while for Pluto to
“native” orbit, but it’s more fun to circle our solar system, doesn’t it? It’s
have this be based on Earth days, 90,474 days.
although when we get to Pluto (yeah, For this script, we’ll want users to
I still consider it a planet), it’s rather enter their birthdays, calculate how
a long time. many days it’s been since they were
Here’s the data, in 24-hour Earth days: born, then simply divide that number
by the “year” on each planet to
Mercury 87.96 calculate their universe ages.
Venus 224.68 I’ve actually written about how
Earth 365.26 to calculate days between a specific
Mars 686.98 date in the past and the current day,
Jupiter 11.862 years but rather than show the exhaustive
Saturn 29.456 years calculation, let’s just lean on the date
Uranus 84.07 years function—more specifically GNU
Neptune 164.81 years date )TS QUITE LIKELY WHAT YOU HAVE
Pluto 247.7 years on your computer already, and you
can find out simply by typing:
We will need to convert the last
five into Earth days, but that’s >
easy: just multiply by 365.26 (to $ date --version
be accurate), which gives us this date (GNU coreutils) 8.23
better reference chart, presented Copyright (C) 2014 Free Software Foundation, Inc.
as variables ready for a script: License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>.
mercury=87.96 This is free software: you are free to change and

venus=224.68 redistribute it. There is NO WARRANTY, to the
earth=365.26 extent permitted by law.
mars=686.98
jupiter=4332.71412 This is on the latest version of
saturn=10759.09856 Ubuntu. Sadly, I’m going to be leaving
uranus=30707.4082 you Mac users who have become

COLUMNS
WORK THE SHELL
accustomed to working with my scripts we’re subtracting the current date from
in the dust this time. Unfortunately, the day in the past, not vice versa.
Mac OS X still ships with the older Flipping the math around in the
POSIX version of date and therefore EQUATION SOLVES THE PROBLEM AND GETS
has no date math available. With the desired result:
GNU date , however, it’s super easy to
calculate the number of days you’ve Born on aug 3 1965 means you've been alive 18354 days.
been alive. Let’s assume you, like me,

were born on August 3, 1965 (my .OW THE REST OF THE SCRIPT IS QUITE EASY
birthday, plus or minus a year or three). particularly since we’ve already translated
How many days have I been alive? each and every planet’s orbital duration
The one-liner: into Earth days. To demonstrate, 18,354
Earth days would make me 26.71
daysalive=$(( ( $(date -ud 'aug 3 1965' +'%s') -
(18354 / 686.98) Martian years old.
´$(date -u +'%s') )/60/60/24 )) Here’s the full script:
This has my birthday hard-coded, $ cat planetaryage.sh
probably not what we want, so #!/bin/sh
instead it could be modified to work
with user input, as a first step toward mercury=87.96;; venus=224.68;; earth=365.26
actually creating a script: mars=686.98;; jupiter=4332.71412;; saturn=10759.09856
uranus=30707.4082;; neptune=60198.5006;; pluto=90474.902
$ cat planetaryage.sh
#!/bin/sh planetaryAge()
daysalive="$(( ( $(date -ud "$*" +'%s') - {
´$(date -u +'%s') )/60/60/24 ))" orbit=$1
echo "Born on $* means you've been alive $daysalive days." planetname=$2
exit 0 planetarydays=$( echo "scale=5;;$daysalive / $orbit"|bc )
$ sh planetaryage.sh aug 3 1965 echo "You are $planetarydays $planetname years old."
Born on aug 3 1965 means you've been alive -18354 days. }
Negative days. It seems like something daysalive="$(( ( $(date -u +'%s') -
you’d get out of an old Night Gallery ´$(date -ud "$*" +'%s') )/60/60/24 ))"
show or perhaps Black Mirror, to be a bit
more contemporary. But why? Because planetaryAge $mercury "Mercury"

COLUMNS
WORK THE SHELL
planetaryAge $venus "Venus" You are 127.24387 Mars years old.

planetaryAge $earth "Earth" You are 20.17534 Jupiter years old.
planetaryAge $mars "Mars" You are 8.12465 Saturn years old.
planetaryAge $jupiter "Jupiter" You are 2.84667 Uranus years old.
planetaryAge $saturn "Saturn" You are 1.45209 Neptune years old.
planetaryAge $uranus "Uranus" You are .96616 Pluto years old.
planetaryAge $neptune "Neptune"
planetaryAge $pluto "Pluto" Ah, it was almost one Plutonian

year ago. It’s simple enough once you
exit 0 know the mathematical capabilities of
GNU date , for sure. Without it, you
It’s actually not too complicated. When definitely can cobble together a script
I run it for my own birthday, I find out that can calculate the number of days
)M QUITE A SPRING CHICKEN ON .EPTUNE since a specified date in the past, but
we’ve done that before!
$ sh planetaryage.sh aug 3 1965
You are 208.66302 Mercury years old. Next Month
You are 81.68951 Venus years old. Now is a golden opportunity, as we head
You are 50.24913 Earth years old. into our next decade of this column, for
You are 26.71693 Mars years old. you to send in some puzzles, ideas or
You are 4.23614 Jupiter years old. topics you’d like me to cover here (send
You are 1.70590 Saturn years old. e-mail to ljeditor@linuxjournal.com).
You are .59770 Uranus years old. Don’t be shy!Q
You are .30489 Neptune years old.
You are .20286 Pluto years old. Dave Taylor has been hacking shell scripts since the dawn of the
computer era. Well, not really, but still, 30 years is a long time!
What about duration since the He’s the author of the popular Wicked Cool Shell Scripts and
signing of the Declaration of Teach Yourself Unix in 24 Hours (new edition just released!).
Independence? Those numbers are He can be found on Twitter as @DaveTaylor and more generally
a bit more interesting: at his tech site: http://www.AskDaveTaylor.com.
$ sh planetaryage.sh july 4 1776

You are 993.79263 Mercury years old. Send comments or feedback via
You are 389.05999 Venus years old. http://www.linuxjournal.com/contact
You are 239.31993 Earth years old. or to ljeditor@linuxjournal.com.

COLUMNS
HACK AND /
Preseeding KYLE RANKIN
Full Disk
Encryption
Automation makes things faster, if you don’t count
all that work ahead of time.
Usually I try to write articles that are automated “OEM” install for a
not aimed at a particular distribution. laptop. The goal was to have an
Although I may give examples automated boot mode that would
assuming a Debian-based distribution, guide users through their OS install
whenever possible, I try to make my and use full-disk encryption by
instructions applicable to everyone. default, but would make the process
This is not going to be one of those as simple as possible for users.
articles. Here, I document a process Normally, unless you are going to
I went through recently with Debian encrypt the entire disk as one big
preseeding (a method of automating partition, the Debian installer makes
a Debian install, like kickstart on Red you jump through a few hoops to set
Hat-based systems) that I found much up disk encryption during an install.
more difficult than it needed to be, In my case, I couldn’t just use the
mostly because documentation was full disk, because I needed to carve
so sparse. In fact, I really found only off a small section of the disk as a
two solid examples to work from in rescue partition to store the OEM
my research, one of which referred install image itself. My end goal was
to the other. to make it so users just had to enter
In this article, I describe how to their passphrase, and it would set
preseed full-disk encryption in a up an unencrypted /boot and rescue
Debian install. This problem came disk partition and an encrypted / and
up as I was trying to create a fully swap. As an additional challenge,

COLUMNS
HACK AND /
My end goal was to make it so users just had

to enter their passphrase, and it would set up
an unencrypted /boot and rescue disk partition
and an encrypted / and swap.
I also wanted to skip the time- Since you need a basic unencrypted
consuming disk-erasing process that /boot partition to load a kernel and
typically happens when you enable prompt the user for a passphrase, I
disk encryption with Debian, since the had to account for both and preserve
disk was going to be blank to start a small 2GB rescue disk partition
with anyway. that already was present on the disk.
Unfortunately, although there is After that, the remaining / and swap
a lot of documentation on how to partitions were encrypted. Here is the
automate ordinary partitioning and partition section of the preseed config:
,6- WITH PRESEEDING ) ACTUALLY WROTE
a whole section on the topic myself d-i partman-auto/method string crypto
in one of my books), I had a hard d-i partman-lvm/device_remove_lvm boolean true
time finding much documentation on d-i partman-lvm/confirm boolean true
how to add encryption to the mix. d-i partman-auto-lvm/guided_size string max
After a lot of research, I finally found d-i partman-auto-lvm/new_vg_name string crypt
two posts (and as I mentioned, one d-i partman-auto/disk string /dev/sda
of them referenced the other) that d-i partman-auto/choose_recipe select root-encrypted
described the magic incantation that d-i partman-auto/expert_recipe string \
would enable this. Unfortunately, the root-encrypted :: \
only supported mode for encrypted 500 500 500 ext3 \
DISKS IN $EBIAN PRESEED REQUIRES THE $primary{ } $bootable{ } \
USE OF ,6- SOMETHING ) CONFIRMED method{ format } format{ } \
later when I read the source code use_filesystem{ } filesystem{ ext4 } \
responsible for this part of the install). mountpoint{ /boot } \
That’s not the end of the world, but it . \
would have been simpler in my mind 2000 2000 2000 linux-swap \
IF IT DIDNT HAVE THAT REQUIREMENT $lvmok{ } lv_name{ swap } \

COLUMNS
HACK AND /
in_vg { crypt } \ most important one tells partman

$primary{ } \ (the preseed partition manager) to
method{ swap } format{ } \ use encryption:
. \
500 10000 1000000000 ext4 \ d-i partman-auto/method string crypto

$lvmok{ } lv_name{ root } \
in_vg { crypt } \ Next, because preseeded encrypted

$primary{ } \ PARTITIONS NEED TO USE ,6- ) MUST ADD
method{ format } format{ } \ ,6- SPECIFIC PRESEED SETTINGS
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ / } \ d-i partman-lvm/device_remove_lvm boolean true
. \ d-i partman-lvm/confirm boolean true
2000 2000 2000 ext4 \ d-i partman-auto-lvm/guided_size string max
$primary{ } \ d-i partman-auto-lvm/new_vg_name string crypt
method{ keep } \
use_filesystem{ } filesystem{ ext4 } \ In the last of these settings, I told

label{ rescuedisk } \ PARTMAN TO CREATE A NEW ,6- VOLUME
. group named crypt that I will use
to store my encrypted partitions.
d-i partman-md/device_remove_md boolean true Further down when I define my swap
d-i partman-basicfilesystems/no_mount_point boolean false and root partitions, you can see
d-i partman-partitioning/confirm_write_new_label boolean true where I defined the logical volumes
d-i partman/choose_partition select finish by name and set what volume group
d-i partman/confirm boolean true they are in:
d-i partman/confirm_nooverwrite boolean true
2000 2000 2000 linux-swap \
If you’ve never worked with $lvmok{ } lv_name{ swap } \
preseeding, this entire section of in_vg { crypt } \
code probably looks incredibly . . .
foreign. As preseeding in general 500 10000 1000000000 ext4 \
is documented well in a number of $lvmok{ } lv_name{ root } \
other places, I’m not going to bother in_vg { crypt } \
breaking down every setting here.

Instead, let me highlight the settings Once these settings were in place,
that matter for disk encryption. The I was able to preseed an install and

COLUMNS
HACK AND /
have disk encryption be almost fully this and do a search on-line, they at
automated, except that the installer least can find my article and the two
prompted me for a passphrase, which other examples and won’t have to
I wanted. burn so much time. Q
The only missing piece to this
automation was that the installer Kyle Rankin is a Sr. Systems Administrator in the San Francisco
started overwriting the existing disk Bay Area and the author of a number of books, including The
with random information. Now, Official Ubuntu Server Book, Knoppix Hacks and Ubuntu Hacks.
there are good reasons why you He is currently the president of the North Bay Linux Users’ Group.
may want to do this before setting
up disk encryption, but in this case,
the disk was blank beforehand, Send comments or feedback via
and I didn’t want to wait the many http://www.linuxjournal.com/contact
hours it might take. Try as I might, or to ljeditor@linuxjournal.com.
no options to preseed this feature
away seemed to work. After poring
through the partman code to find
the magic option, I finally resorted LINUX JOURNAL
to patching the partman-crypto
script on the fly in the middle of
for iPad and iPhone
the install so that it skipped the
erase process:
d-i partman/early_command \
string sed -i.bak 's/-f $id\/skip_erase/-d $id/g'
/lib/partman/lib/crypto-base.sh
This is an ugly hack indeed, but

it was the only way I was able to
find that worked. With that in
place, I was able have an automated
partitioning recipe with full-disk
encryption that skipped the disk-
erasing section. My hope is that the
next time other people need to do http://www.linuxjournal.com/ios

COLUMNS
THE OPEN-SOURCE CLASSROOM
Profiles and SHAWN POWERS
RC Files
Confused by profiles and bashrc? Read on!
I love Linux, and if you’re reading terminal shell, you’re prompted for
this, chances are you do too. To be a user name and password. Other
honest though, some aspects of the times, you just click on the terminal
Linux environment are confusing. icon, and you’re presented with a
Near the top of the list for me is terminal already logged in. You’ll
the profile system. Conceptually, most often experience this when
it’s simple. There are system-wide using a GUI desktop environment.
settings that all users inherit, and Basically, if you’re already logged
then there are individual settings in to your Linux desktop, and you
people can set on their own. The open a terminal window, it’s an
problem comes when different interactive shell.
distributions handle profiles in It doesn’t have to be inside a
different ways, and the concept graphical desktop environment,
of login shells versus interactive however. If you ssh in to a
shells comes into play. Usually, it’s remote server, you’re prompted
not something Linux users worry for a user name and password
about. But, when you need to (thus, a login shell). If you then
make a change, it can be extremely type bash from inside that SSH
frustrating to figure out what is session, you’re starting a brand-
loaded in what order, and which is new terminal, but this time, it’s
seen by login shells only, and so on. an interactive shell (notice you’re
not prompted for a password).
Login Shells Why it matters is something I’ll
First, let me clarify what I mean by talk about a little later, but for
login shells. You’ve probably noticed comprehension sake, just remember
that sometimes in order to get to a that if you’re prompted for a user

COLUMNS
Having a folder to add custom scripts is important,

because if you have system-wide changes you’d
like added to everyone’s login shell, adding
commands to the /etc/profile file is dangerous.
name and password, it’s most likely should have set.

a login shell. If you go directly to Step 2: the /etc/profile script
a bash prompt, it’s most likely an usually ends by calling any shell
interactive shell. The one fairly scripts in the /etc/profile.d folder and
common exception to this is if executing them as well. Often it will
you’ve set up SSH keys to log in run only shell scripts in /etc/profile.d
automatically. In that case, even that end with a .sh extension, so
though you aren’t prompted for a look at the /etc/profile script to see
user name and password, it’s still a how files should be formatted to
login shell. It’s a pretty safe bet that run properly. Having a folder to add
if you’re using SSH to log in, it’s a custom scripts is important, because
login shell. if you have system-wide changes
you’d like added to everyone’s login
The Login Shell Process shell, adding commands to the
The login shell process is far more /etc/profile file is dangerous. Any
complicated than interactive shells, system updates affecting /etc/profile
so I am going to go over that will overwrite your changes. If you
process first. I’m assuming your simply add a custom file into the
users have a bash shell assigned in /etc/profile.d folder, it will be read
their /etc/passwd files. It’s the most by the updated /etc/profile script
common shell for users to have, so even if it’s updated.
it makes sense to be familiar with Step 3: the /etc/profile script also
its nuances. executes the user’s personal profile.
Step 1: when you authenticate in This part is a little messy, as the user
to a login shell, the system looks profile might be called different
for a file called /etc/profile. That things depending on distribution
file is a shell script that assigns a and/or user customization. In
few environment variables all users general, the system will try loading

COLUMNS
the profile by name in this order: is the .bashrc file stored in the user
directory. This is another script—this
Q .bash_profile one called from the .profile script in
Step 3. Note that if you customize
Q .bash_login your user profile settings, you’ll want
to make sure whatever profile file
Q .profile you use actually calls the .bashrc
script. It’s inside the .bashrc script
If it finds a file with that name where personal settings like a custom
in the user’s home directory, it prompt and color settings go, along
executes it and stops. This means with command aliases you might
if you have a .bash_profile and want to set (more on those later).
.profile in your home directory, only Step 5: this step doesn’t really
the .bash_profile will be executed. take place after Step 4; rather, it
This is useful to know if you want sort of branches off at Step 1. The
to customize your profile, but /etc/profile script starts the process
don’t want to make changes to the for loading user profiles, but it also
original user profile assigned to you. kicks off the process for executing
By default in Ubuntu, every user has the system-wide bashrc file. Here
a .profile file, but not .bash_profile again various distributions name
or .bash_login. So if you want to this file differently, but it’s generally
customize your profile, simply copy either a file called /etc/bashrc or
the .profile in your home directory /etc/bash.bashrc. In the case of
to a file called .bash_profile, and Ubuntu, it’s /etc/bash.bashrc, but
make any changes you want to historically, it’s often /etc/bashrc.
.bash_profile. Doing that will leave Note that unlike the user’s .bashrc
your original .profile intact and still file, the system-wide bashrc file does
will allow you to customize to your not start with a period.
heart’s content. Just remember, if To add insult to injury, some
you create an empty .bash_profile, systems don’t actually execute the
the system will see that as your system-wide bashrc file for login
profile of choice and ignore your shells, so if you don’t see it called
.profile file completely! in the /etc/profile script, that means
Step 4: finally, the last step along it’s not going to execute for login
the login shell order of operations shells. For the most part, however,

COLUMNS
the /etc/profile on the majority of system-wide or in the user directory.

distributions does indeed call the However, because an interactive
system-wide bashrc file. Since you shell is a “child” process of the
know the order with which profiles login shell used to log in initially
are loaded, you can investigate (either via GUI or SSH), it inherits all
on your own system to see what the profile information that initial
is actually loaded during the login LOGIN SHELL ACQUIRED 3O ALTHOUGH
shell startup. both the initial login shell and the
“child” interactive shell have the
Interactive Shell Process same profile information loaded,
An interactive shell has a far simpler the important distinction is that
startup procedure. When a person interactive shells don’t reload the
opens an interactive shell (one that profile information. They never look
doesn’t authenticate a user name or at the profile scripts, so whatever
password), the following steps occur. information was loaded by that
Step 1: the /etc/bashrc or /etc/ initial login script is all they have
bash.bashrc file is executed. This access to. (This distinction will be
takes place whether or not it’s more important when you see what
referenced in /etc/profile. While a the scripts actually do.)
login shell automatically starts the
/etc/profile script, an interactive shell What Do Profiles Do?
automatically starts the /etc/bashrc First, a disclaimer: I can spell out
(or /etc/bash.bashrc) script. only what is generally done with
Step 2: the users’ .bashrc from profiles and bashrc scripts. It’s
their home directory is executed. certainly possible for a person to
Again, like the system-wide bashrc change what is done by customizing
file, this isn’t called from a user either profiles or bashrc scripts.
profile; rather, it’s executed directly Generally, it’s good practice to stick
by the interactive shell. So even to the standards.
if you’ve erased the reference to Profiles mainly are used to load
.bashrc from the .profile script, an environment variables. Since profiles
interactive shell still will execute it. are loaded by login shells, and
And, that’s it! An interactive login shells are the initial entry
shell doesn’t look for any point into a system, that’s the time
profile information at all, either when setting up the environment

COLUMNS
makes the most sense. One of the prefer a specific color scheme, the
biggest environment variables is the bashrc system is where that would
PATH variable. When a login shell be set. Much like the profile system,
is initiated, the PATH is set. Other the user’s .bashrc file overrides the
environment variables also can be system-wide bashrc (or bash.bashrc,
set in the system-wide profile or again see above) settings. That
individual user profiles, but just means you can customize the
know that the profile system is behavior of the command line
where most variables are set. however you like without affecting
The order with which profile other users on the system.
information is loaded is very The most common customization
important, because if you want to inside the .bashrc file is to add
override the system-wide default aliases. An alias is sort of like text
profile information, you can do so expansion, in that it substitutes
by specifying environment variables your defined alias with whatever
in your personal user profile script. command you specify. For example,
For instance, the PATH variable here’s a snippet from a .bashrc file
is usually modified by the user’s in the user’s folder:
profile script on login. Usually, the
.profile (or .bash_profile, etc., see alias ll='ls -alF'
above) script will add ~/bin to the alias la='ls -A'
PATH variable if users have their alias l='ls -CF'
own bin folder inside their home
directory. Because user profiles The aliases make it so that if
are loaded after the system-wide the user types ll on the command
profile, user settings take precedent line, the system will execute
and override system-wide settings. ls -alF instead. It’s a great way
to make shortcuts for commands
What Do RC Files Do? with cryptic options or shortcuts
Again, this is a generalization, for commands you type often.
but the system-wide bashrc file Although I’m not suggesting
and then the individual user’s tomfoolery, .bashrc aliases are also
.bashrc script usually set personal a great way to prank your fellow
preferences for the command line. users if they leave their system
If you want a custom prompt, or logged in. Say you create an alias

COLUMNS
like this: only way to make permanent changes.

Also, although I went over it
alias ls='echo "Deleting all files..."' already, I want to reiterate that while
the system-wide profile (/etc/profile)
Then, every time they type ls , and the user profile generally call the
they’ll be in for a little (innocent) bashrc scripts, they don’t have to.
surprise! Yes, it’s very easy to do If you make changes to your profile
nefarious pranks with aliases, but settings, it’s possible that your login
since we all log out when we leave shells will behave very differently
our workstation, we shouldn’t ever from your interactive shells. If that’s
have to worry about it, right?! your goal, great, but usually you want
to make sure your login shells also
Some Gotchas execute the bashrc stuff, since that
Understanding how shells work really information is what makes the user
makes troubleshooting a lot easier. experience more useful.
You’ve probably already realized a few Finally, I want to add that the
things, but they’re worth mentioning. best way to understand and learn
If you make changes to any of the about profiles and RC files is to
profile scripts, those changes won’t play with your system. Learning how
be recognized until you start a new to manipulate your settings is not
login shell. The same is true for .bashrc only educational, but it can make
changes, but since you easily can close your computing experience much
an interactive shell and start a new one, more convenient. Q
.bashrc changes are easier to activate.
One of the main problems regarding Shawn Powers is the Associate Editor for Linux Journal.
profile loading is that if you make He’s also the Gadget Guy for LinuxJournal.com, and he has an
a change to environment variable interesting collection of vintage Garfield coffee mugs. Don’t let
settings, like the PATH variable, you’ll his silly hairdo fool you, he’s a pretty ordinary guy and can be
actually have to log out and log back reached via e-mail at shawn@linuxjournal.com. Or, swing by
in to test your changes. You certainly the #linuxjournal IRC channel on Freenode.net.
can set a path variable on an interactive
shell, but remember, any new
interactive shells will inherit the original Send comments or feedback via
login shell’s profile settings, so often http://www.linuxjournal.com/contact
logging out and back in is really the or to ljeditor@linuxjournal.com.

NEW PRODUCTS
DJI Manifold
Canonical’s Ubuntu operating system serves as the “brains” for
DJI Manifold, a new, high-performance embedded computer
for drones that reduces processing time and optimizes real-
time data analysis. Utilizing DJI’s Onboard SDK, the Manifold
is a user-friendly system that enables developers to create
more powerful professional applications that leverage aerial and ground technologies to solve complex
problems. Fully compatible with DJI’s Matrice 100 drone, the Manifold is also compatible with third-party
sensors and enables developers to connect a wide variety of onboard devices, such as infrared cameras,
ATMOSPHERIC RESEARCH DEVICES AND GEOGRAPHICAL SURVEYING EQUIPMENT "ECAUSE THE -ANIFOLD COMPUTER
both collects and analyzes data in the air, it provides an efficient solution for developers in need of
TIME SENSITIVE INFORMATION 2ELEVANT TECH SPECS INCLUDE 5BUNTU ,43 VERSION QUAD CORE !2-
#ORTEX ! PROCESSOR .6)$)! +EPLER BASED '05 AND SUPPORT FOR #5$! /PEN#6 AND 2/3
http://www.dji.com
Donald Simpson’s Beginning Docker

(Packt Publishing)
If you’re a developer seeking to learn how to deploy applications in
containers using the open-source Docker, you have a new learning tool at
your disposal. Beginning Docker, taught by automation build engineer and
DevOps consultant Donald Simpson and published by Packt Publishing, is
a two-hour-long, hands-on video course packed with practical examples
TO GET ONE STARTED WITH $OCKER AND CREATE AMAZING APPLICATIONS %QUIPPED WITH BASIC KNOWLEDGE
of Linux, viewers of Beginning Docker will learn how Docker works, how to set it up and how to
get started on leveraging its benefits. From there, viewers create and share Docker images, install
Docker on their own machines, learn to manage it effectively, and then progress to creating and
publishing a custom application. Advanced topics include Docker containers, volumes, mounts,
ports, linking and constraining containers, the Docker Web API, handling of complex automation
processes with Shipyard and the creation of a mini-Heroku PaaS with slugbuilder and slugrunner.
Packt describes the format of Beginning Docker as an easy-to-follow and structured video tutorial
with practical examples of Docker to help viewers get to grips with each and every aspect.
http://www.packtpub.com

NEW PRODUCTS
Rogue Wave Software’s CodeDynamics

Helped on by a nudge from Rogue Wave Software’s CodeDynamics tool, large-scale data
modelling and analytics technologies traditionally reserved for HPC are making their way
into new enterprise and industrial applications. These big data solutions in financial,
ENERGY SCIENCE GOVERNMENT AND OTHER COMMERCIAL SERVICES REQUIRE THE SAME ELEMENTS
as HPC: extremely fast systems, highly optimized code and innovative algorithms. Enter
CodeDynamics, Rogue Wave’s next-generation dynamic analysis tool for data-intensive
commercial applications that expands the reach of multithreaded debugging. CodeDynamics looks
at complex C and C++ applications at execution time to help identify and correct bugs, memory
ISSUES AND CRASHES CUTTING hRIGHT TO THE CHASEv TO IDENTIFYING CAUSES QUICKLY %NTERPRISES THAT DEMAND
PERFORMANCE SCALABILITY AND HIGH AVAILABILITY WILL FIND VALUE IN THE DEEP THREAD CONTROL UNIQUE REVERSE
debugging and advanced data visualization features of CodeDynamics, adds Rogue Wave. An
additional innovation in CodeDynamics is the ReplayEngine feature, built to simplify the troubleshooting
process. By recording and saving program execution, ReplayEngine allows developers to work back
from a failure, error or crash to find the root cause without repetitive restarts and stops, and it allows
developers to store the recording to investigate the error at any time.
http://www.roguewave.com
IBM’s API Harmony

IBM prefaced the announcement for its new API Harmony platform by
stressing not just the critical importance of APIs for organizations to become more “cognitive” but
also the value of the “API Economy”, destined to be worth $2.2 trillion by 2018. To exploit the vast
potential of APIs—that is, application programming interfaces—while navigating their risks, Big Blue
presents API Harmony, a cloud service on the Bluemix Development Platform that acts as a matchmaker
of APIs for developers and IT managers to facilitate the process of building new applications. Armed
with advanced cognitive technologies like intelligent mapping and graph technology, API Harmony
PROVIDES A UNIQUE DEVELOPER EXPERIENCE TO ANTICIPATE WHAT A DEVELOPER WILL REQUIRE TO BUILD NEW APPS
make recommendations, show API relationships and identify what is missing. IBM adds that when
core information assets are packaged as APIs and shared or sold, enterprises build awareness, increase
customer satisfaction through more personalized services and expand partner networks. API Harmony is
one of many services and solutions from IBM that provide the foundation for clouds to behave as one,
leading to more consistent cloud integration regardless of the cloud infrastructure.
http://www.ibm.com/apieconomy

NEW PRODUCTS
JetBrains Toolbox
In this space, I typically cover about
eight new products for your reading
pleasure. This month, however, I feature
more than double the normal output,
thanks to a “big day” of updates from
JetBrains s.r.o. The tool developer
simultaneously upgraded the nine elements in its JetBrains Toolbox, thus smashing the Linux Journal
New Products record for most products announced in a single issue. These nine elements include
IntelliJ IDEA 15 IDE for Java, PhpStorm 10 IDE for Java, WebStorm 11 IDE for JavaScript, PyCharm
5 IDE for Python, AppCode 3.3 IDE for Objective-C on Mac OS X, CLion 1.2 cross-platform IDE for
# AND # 2UBY-INE )$% FOR 2UBY AND 2AILS X$"% )$% FOR $"!S AND 31, $EVELOPERS AND
2E3HARPER 5LTIMATE PRODUCTIVITY TOOL FOR 6ISUAL 3TUDIO )N ADDITION TO THE PRODUCT IMPROVEMENTS
for each tool, JetBrains added a new “All Products” pack that allows customers to use any of the
above products according to their current needs.
http://www.jetbrains.com
Undo Software to Deliver Support

for 64-bit ARM Devices
Debugging, asserts Undo Software, is the number one challenge when moving existing code to new
architectures like the 64-bit ARM v8-A. To simplify porting code to the ARM v8-A, Undo Software Ltd.
and ARM have teamed up to produce a portfolio of advanced Linux and Android reversible debugging
tools—the most recent and notable of which is the Undo Software’s Live Recorder. Live Recorder’s new
“software-implemented trace” debugging technology helps simplify porting code from alternative
hardware architectures by enabling Linux and Android programs to make a detailed recording of
themselves while they are running. The recording, executed in a highly compressed and efficient
way, contains everything needed for a developer to debug an exact copy of the bug as it occurred in
production. This includes everything a program does, such as every memory access made and every
instruction executed. This information can be used to run and step their programs backward and
FORWARD IN TIME ENABLING DEVELOPERS TO FIX BUGS MORE QUICKLY ,IVE 2ECORDER DELIVERS PARTICULAR BENEFITS
to telecoms, IoT, enterprise server, HPC, mobile and automotive industries, the sectors most advanced in
porting existing software to the 64-bit ARM architecture.
http://www.undo-software.com

NEW PRODUCTS
1248’s DevicePilot
Unlike Web or smartphone apps, connected Internet of Things (IoT) devices must be
deployed into the physical world, where lots of things can go wrong. To overcome
physical-world barriers that stand in the way of effective IoT at scale, IoT specialist
1248 unveiled DevicePilot, a new as-a-service solution for managing the growing IoT
ecosystem. DevicePilot continuously monitors and manages connected devices over
their complete life cycles and presents a simple dashboard showing how many devices
have been deployed, where and by whom, and how many are not working and why.
DevicePilot’s automatic asset management, monitoring and lifetime support enable
scaling projects from pilot stage to deployment with thousands or even millions of
devices with universal coverage, from applications as variable as smart energy to smart
homes and cities to transport systems, as well as industrial monitoring and control. DevicePilot is
integrated with the ARM mbed IoT Device Platform, based on open standards, technology and services
to accelerate wider adoption of IoT systems at scale. The goal of 1248 is to fill one of the few remaining
GAPS IN THE SET OF SERVICES REQUIRED FOR SUCCESSFUL )O4 DEPLOYMENTTHAT IS IN DEVICE MANAGEMENT
http://1248.io
SUSE OpenStack Cloud

Based on the OpenStack release Liberty, the upgraded SUSE
OpenStack Cloud 6 offers enterprise customers the latest
features that further ease transition of business-critical
applications and data to the cloud. SUSE OpenStack Cloud 6 is
SUSE’s solution for building Infrastructure-as-a-Service private
clouds. In addition to high-availability enhancements and non-disruptive upgrades to future
OpenStack releases, this new version also adds Docker for containerized applications and IBM z
3YSTEMS MAINFRAME SUPPORT TO EXISTING SUPPORT FOR 8EN +6- (YPER 6 AND 6-WARE HYPERVISOR
options. Finally, full support for OpenStack Manila provides direct access to the performance,
scalability and management of the open-source Manila shared filesystem service.
http://www.suse.com
Please send information about releases of Linux-related products to newproducts@linuxjournal.com or

New Products c/o Linux Journal, PO Box 980985, Houston, TX 77098. Submissions are edited for length and content.

FEATURE Secure File Transfer
SECURE
FILE TRANSFER
How to improve file transfer security
with RFC 1867, thttpd and Stunnel.
CHARLES FISHER

F
ile transfer between Linux performance through the years in
systems (and perhaps all the FTP server software that is
POSIX systems in general) is commonly bundled with Linux
in some ways a neglected subject. (https://security.appspot.com/
The arcane protocols in common vsftpd.html#security). It remains
use are far from secure, and the easy to configure FTP clients for batch
SSH replacements offer too much activity with automatic logins:
power and complexity. Servers holding
highly sensitive data (such as credit echo machine a_server.com login YourName password

card numbers, SSNs, birthdates and á_Password >> ~/.netrc
so on) often must accept file transfers, chmod 600 ~/.netrc
but greatly restrict remote visibility echo -e 'ls -l \n quit' | ftp a_server.com
and administration, which is hard

with the well known tools. Unfortunately, this is a terrible idea
File transfers with RFC 1867 that gets progressively worse with the
(https://www.ietf.org/rfc/rfc1867.txt) passage of time:
can offer a number of benefits over
most other methods: the highest Q The login, password and file
security and optional encryption, payload are all sent in clear
ALL WITHOUT REQUIRING ENTRIES IN text over the wire in the normal
/etc/passwd or other credentials configuration, and there are many
for the operating system. utilities to capture them that might
The tools I cover in this article to be used over an untrusted network.
implement this protocol are sthttpd,
an upload CGI utility, stunnel and Q Classic FTP servers listening on port
curl. The examples here were 21 must run as root. If attackers
developed on Oracle Linux 7.1, but find and exploit a weakness, your
most of the code is portable and OS belongs to them.
should run on other platforms with
minimal changes (with the exception Q In “active” FTP, the client and
of the systemd configuration). server switch roles in running the
connect() and listen() system
Why Not FTP? calls. This causes the TCP connections
There have been substantial to open in both directions,
improvements in security and introducing problems for firewalls.

Q Unless the FTP server supports and also data connections. You'll need a client with
chroot() and it is individually SSL support too. NOTE!! Beware enabling this option.
and specifically configured for Only enable it if you need it. vsftpd can make no
a target user, that user is able guarantees about the security of the OpenSSL libraries.

to fetch recursively all accessible By enabling this option, you are declaring that you
files on the system that have trust the security of your installed OpenSSL library.
world-read permission.
The reason for the above warning
Q An FTP account created for a is that because the FTP server runs
few files can give visibility to just as root, it exposes the encryption
about everything. Most modern library to remote connections with the
FTP clients allow such recursive highest system privilege. There have
TRANSFERS !N &40 USER REQUIRES AN been many, many encryption security
entry in /etc/passwd on the server flaws through the years, and this
that creates an OS account. If not configuration is somewhat dangerous.
properly managed, this allows the The OpenSSH suite of
remote user to log in to a shell or communication utilities includes
otherwise gain unwanted access. “sftp” clients and servers, but
THIS ALSO REQUIRES AN ACCOUNT ON
Q Password aging often is mandated the operating system and special
in high-security environments, key installation for batch use. The
REQUIRING SYNCHRONIZED PASSWORD recommended best practice for key
changes on the client and server HANDLING REQUIRES PASSWORDS AND THE
(usually after a failed overnight use of an agent:
batch run).
Our recommended method for
Later revisions to the FTP protocol best security with unattended
do add TLS/SSL encryption capabilities, SSH operation is public-key
but it is unwise to implement them: authentication with keys stored
in an agent....The agent method
man vsftpd.conf | col -b | awk '/^[ ]*ssl_enable/,/^$/' does have a down side: the system
ssl_enable can’t continue unattended after
If enabled, and vsftpd was compiled against OpenSSL, a reboot. When the host comes
vsftpd will support secure connections via SSL. This up again automatically, the batch
applies to the control connection (including login) jobs won’t have their keys until

someone shows up to restart the and configured by default.
agent and provide the passphrases None of the above programs
to load the keys.—SSH, the Secure work well for secure batch file copy
Shell, 2nd Edition, Daniel J. when receiving files from untrusted
Barrett, Richard E. Silverman and sources, and for these reasons, let’s
Robert G. Byrnes. turn to RFC 1867.
Those who blindly rush from FTP thttpd in a chroot()

to sftp due to security pressures do RFC 1867 is the specification behind
not understand the complexities of the “file upload gadget” found on
key generation, the ssh-agent and Web pages. The HTML to implement
ssh-add. Forcing such sophisticated the gadget is relatively simple:
utilities on a general population that
is attempting to migrate away from <form action="script.cgi" enctype="multipart/form-data"
FTP is sure to end badly. method="post">
OpenSSH also extends the <input type="file" name="Whatever">
ability to run a shell to the client <input type="submit" value="Upload">
in the default configuration. It </form>
is possible to constrain a user to

file transfers only and configure 6ARIOUS BROWSERS RENDER THE
for a higher-security chroot(), but gadget with a slightly different
extensive modifications to the server appearance, but the function is the
configuration must be performed to same (Figures 1–3).
implement this. The main focus of
SSH is secure interactive login—file
transfers are a sideline. The lack
of “anonymous” sftp or keyed file Figure 1. Google Chrome
dropoff highlight this (lack of) focus.
The classic Berkeley R-Utilities
include an rcp program for remote
file copy. This does eliminate the Figure 2. Microsoft Internet Explorer
clear-text password, but improves
little else. The use of these utilities
is highly discouraged in modern
systems, and they are not installed Figure 3. Mozilla Firefox

For this article, I will be using the suffer from a bug, turning a bug
“curl” non-graphical, command-line into a nuisance rather than a full
tool to perform file transfers using this catastrophe. No root were harmed
protocol. Since the RFC 1867 protocol is during this audit as far as we know.
implemented over HTTP, a Web server is
needed. The server software choice here The common Web servers on Linux,
will be unconventional, for I’m going to Apache and Nginx repeatedly have
REQUIRE NATIVE SUPPORT FOR THE CHROOT refused to implement native chroot()
system call, which isolates running security (http://www.openbsd.org/
processes in the filesystem tree. This papers/httpd-asiabsdcon2015.pdf):
prevents access to powerful programs in
/sbin and any other sensitive data stored OpenBSD has run its Web servers
in restricted locations. in a chroot for many years; Apache
Liberal use of chroot() and privilege and nginx have been patched to
separation recently saved OpenBSD’s run chroot’ed by default. These
new mail system from disaster in a code patches have never been accepted
audit (http://undeadly.org/cgi?action= by upstream, but yet they provide a
article&sid=20151013161745): significant benefit.
First of all, on the positive side, Although this refusal precludes the
privileges separation, chrooting and use of Apache and Nginx in high-
the message passing design have security applications, the recently
proven fairly efficient at protecting updated sthttpd Web server
us from a complete disaster. (http://opensource.dyc.edu/sthttpd)
[The] Worst attacks resulted in does offer this capability. thttpd lacks
[the] unprivileged process being many modern features (FastCGI, SPDY
compromised, the privileged and SSL/TLS), but the native chroot()
process remained untouched, so trumps the disadvantages. Here are
DID THE QUEUE PROCESS WHICH RUNS the steps to download and install it:
as a separate user too, preventing
data loss....This is good news, wget ftp://opensource.dyc.edu/pub/sthttpd/sthttpd-2.27.0.tar.gz
we’re not perfect and bugs will tar xvzf sthttpd-2.27.0.tar.gz
creep in, but we know that these cd sthttpd-2.27.0/
lines of defense work, and they do
reduce considerably how we will ./configure

make install exec_prefix=/home/jail If you point a Web browser at your
machine (first try http://localhost—
mkdir /home/jail/etc your firewall rules might prevent you
mkdir /home/jail/logs from using a remote browser), you
mkdir /home/jail/htdocs should see a directory listing:
mkdir /home/jail/upload
chown nobody:nobody /home/jail/logs /home/jail/upload Index of /

mode links bytes last-changed name
echo 'port=80 dr-x 2 6 Oct 22 22:08 ./
dir=/home/jail dr-x 6 51 Oct 22 22:08 ../
chroot
data_dir=/htdocs If you wish, you can explore

#data_dir=/home/jail/httpd/htdocs your new chroot() environment by
user=nobody downloading a copy of BusyBox.
cgipat=**.xyz BusyBox is a statically linked collection
pidfile=/home/jail/logs/thttpd.pid of “miniature” UNIX/POSIX utilities,
logfile=/home/jail/logs/thttpd.log' > /home/jail/etc/thttpd.conf with several tools specific to Linux.
When BusyBox binaries are prepared
Note above the cgipat=**.xyz in such a way that they have no
for executing programs that adhere external library linkage, they are
to the Common Gateway Interface perfect for running inside a chroot():
(https://en.wikipedia.org/wiki/
Common_Gateway_Interface). cd /home/jail/sbin
The thttpd documentation mentions wget http://busybox.net/downloads/binaries/busybox-x86_64
using the conventional .cgi chmod 755 busybox-x86_64
extension, but I suggest you pick
your own random extension and ln -s busybox-x86_64 sh
rename any CGI applications that
you deploy to make them harder to cd ../htdocs
find and exploit by an attacker. echo 'Keep out! This means you!' > index.html
After you have installed the thttpd
Web server, you can start a copy with
the following command: echo '#!/sbin/sh
/home/jail/sbin/thttpd -C /home/jail/etc/thttpd.conf echo Content-type: text/plain

echo "" HTTP_ACCEPT=text/html,application/xhtml+xml,application/
/sbin/busybox-x86_64 env ´xml;;q=0.9,*/*;;q=0.8
echo "---" HTTP_HOST=localhost
/sbin/busybox-x86_64 id SERVER_SOFTWARE=thttpd/2.27.0 Oct 3, 2014
echo "---" PATH=/usr/local/bin:/usr/ucb:/bin:/usr/bin
/sbin/busybox-x86_64 ls -l / HTTP_ACCEPT_LANGUAGE=en-US,en;;q=0.5
echo "---" SERVER_PROTOCOL=HTTP/1.1
/sbin/busybox-x86_64' > script.xyz HTTP_ACCEPT_ENCODING=gzip, deflate
REQUEST_METHOD=GET
chmod 755 script.xyz PWD=/htdocs
SERVER_PORT=80
Notice first that an index.html blocks SCRIPT_NAME=/script.xyz
the directory list. Ensure that your CGI SERVER_NAME=localhost.localdomain
applications are protected this way, ---
so they are not seen unless you have uid=99 gid=99 groups=99
chosen to expose them as a <FORM> ---
action. Also observe that a softlink was total 0
created from /sbin/busybox-x86_64 drwxr-xr-x 2 0 0 24 Oct 22 22:08 etc
to /sbin/sh. Calling BusyBox with the drwxr-xr-x 2 0 0 40 Oct 24 15:03 htdocs
link changes the program’s behavior drwxr-xr-x 2 0 0 40 Oct 22 22:10 logs
and turns it into a Bourne shell. The drwxr-xr-x 2 0 0 97 Oct 24 15:02 sbin
program examines $argv[0] , and if ---
the contents match an “applet” that BusyBox v1.24.0.git (2015-10-04 23:30:51 GMT) multi-call binary.
has been compiled into it, BusyBox BusyBox is copyrighted by many authors between 1998-2015.
executes the applet directly. Licensed under GPLv2. See source distribution for detailed
If you now load http://localhost/ copyright notices.
script.xyz with your browser, the shell
script will run, and you should see: Usage: busybox [function [arguments]...]
or: busybox --list[-full]
GATEWAY_INTERFACE=CGI/1.1 or: busybox --install [-s] [DIR]
SHLVL=1 or: function [arguments]...
REMOTE_ADDR=::1
HTTP_USER_AGENT=Mozilla/5.0 (X11;; Linux x86_64;; rv:38.0) BusyBox is a multi-call binary that combines many common
´Gecko/20100101 Firefox/38.0 Unix utilities into a single executable. Most people will
CGI_PATTERN=**.xyz create a link to busybox for each function they wish to

use and BusyBox will act like whatever it was invoked as. rpm2cpio, rtcwake, run-parts, runsv, runsvdir, rx, script,
scriptreplay, sed, sendmail, seq, setarch, setconsole, setfont,
Currently defined functions: setkeycodes, setlogcons, setserial, setsid, setuidgid, sh,
[, [[, acpid, add-shell, addgroup, adduser, adjtimex, arp, sha1sum, sha256sum, sha3sum, sha512sum, showkey, shuf, slattach,
arping, ash, awk, base64, basename, beep, blkid, blockdev, sleep, smemcap, softlimit, sort, split, start-stop-daemon, stat,
bootchartd, bunzip2, bzcat, bzip2, cal, cat, catv, chat, strings, stty, su, sulogin, sum, sv, svlogd, swapoff, swapon,
chattr, chgrp, chmod, chown, chpasswd, chpst, chroot, chrt, switch_root, sync, sysctl, syslogd, tac, tail, tar, tcpsvd, tee,
chvt, cksum, clear, cmp, comm, conspy, cp, cpio, crond, telnet, telnetd, test, tftp, tftpd, time, timeout, top, touch,
crontab, cryptpw, cttyhack, cut, date, dc, dd, deallocvt, tr, traceroute, traceroute6, true, truncate, tty, ttysize,
delgroup, deluser, depmod, devmem, df, dhcprelay, diff, tunctl, ubiattach, ubidetach, ubimkvol, ubirmvol, ubirsvol,
dirname, dmesg, dnsd, dnsdomainname, dos2unix, du, dumpkmap, ubiupdatevol, udhcpc, udhcpd, udpsvd, uevent, umount, uname,
dumpleases, echo, ed, egrep, eject, env, envdir, envuidgid, unexpand, uniq, unix2dos, unlink, unlzma, unlzop, unxz, unzip,
ether-wake, expand, expr, fakeidentd, false, fatattr, fbset, uptime, usleep, uudecode, uuencode, vconfig, vi, vlock,
fbsplash, fdflush, fdformat, fdisk, fgconsole, fgrep, find, volname, watch, watchdog, wc, wget, which, whoami, whois, xargs,
findfs, flock, fold, free, freeramdisk, fsck, fsck.minix, xz, xzcat, yes, zcat, zcip
fstrim, fsync, ftpd, ftpget, ftpput, fuser, getopt, getty,
grep, groups, gunzip, gzip, halt, hd, hdparm, head, hexdump, A few things to point out regarding
hostid, hostname, httpd, hush, hwclock, i2cdetect, i2cdump, each section above:
i2cget, i2cset, id, ifconfig, ifdown, ifenslave, ifup, inetd,
init, insmod, install, ionice, iostat, ip, ipaddr, ipcalc, 1. The environment in the first
ipcrm, ipcs, iplink, iproute, iprule, iptunnel, kbd_mode, section above will include a
kill, killall, killall5, klogd, less, linux32, linux64, linuxrc, QUERY_STRING if you have
ln, loadfont, loadkmap, logger, login, logname, logread, referenced it from a GET-method
losetup, lpd, lpq, lpr, ls, lsattr, lsmod, lsof, lspci, lsusb, form—that is, if you append
lzcat, lzma, lzop, lzopcat, makedevs, makemime, man, md5sum, ?abc=123 to the URL, you will
mdev, mesg, microcom, mkdir, mkdosfs, mke2fs, mkfifo, see QUERY_STRING=abc=123 as
mkfs.ext2, mkfs.minix, mkfs.vfat, mknod, mkpasswd, mkswap, standard GET-method parameters.
mktemp, modinfo, modprobe, more, mount, mountpoint, mpstat,
mt, mv, nameif, nanddump, nandwrite, nbd-client, nc, netstat, 2. User 99 above actually is defined
nice, nmeter, nohup, nslookup, ntpd, od, openvt, passwd, patch, as nobody in the local /etc/passwd
pgrep, pidof, ping, ping6, pipe_progress, pivot_root, pkill, on the test system. Because there is
pmap, popmaildir, poweroff, powertop, printenv, printf, ps, no /etc/passwd file in the chroot(),
pscan, pstree, pwd, pwdx, raidautorun, rdate, rdev, readahead, all user IDs will be expressed
readlink, readprofile, realpath, reboot, reformime, remove-shell,
numerically. If you want users to
renice, reset, resize, rev, rm, rmdir, rmmod, route, rpm, resolve to names for some reason,

define those users in a separate The recent “Towelroot” vulnerability

passwd file copy in the jail. demonstrated an ordinary C program
compiled into a binary executable with
3. It is obvious that the root directory no special permissions that was able
above is confined within the jail. to escalate privilege to root on a great
These files also are resolving to many Linux systems by exploiting a
numeric ownership—if an entry for mutex bug. If your jail includes the
root is placed in the passwd jail ability to download a binary image and
file, named owners will appear. mark it executable, such a flaw could
allow an attacker to smash out of the
BusyBox is useful for exploring a jail and take ownership of your system.
chroot(), but it should not be left on Beware of providing such tools.
a production server, as it introduces If you would like to copy utilities
far too much power. This is confirmed from your host operating system for
on the thttpd Web site with words of use in the jail, you can use the ldd
warning on the contents of the jail command to find their shared object
(http://www.acme.com/software/ dependencies. For example, to move a
thttpd/notes.html): functional copy of GNU AWK into the
jail, examine the dependent objects:
Also: it is actually possible to
break out of chroot jail. A process # ldd /bin/gawk
running as root, either via a setuid linux-vdso.so.1 => (0x00007ffe9f488000)
program or some security hole, can libdl.so.2 => /lib64/libdl.so.2 (0x00007f7033e38000)
change its own chroot tree to the libm.so.6 => /lib64/libm.so.6 (0x00007f7033b36000)
next higher directory, repeating as libc.so.6 => /lib64/libc.so.6 (0x00007f7033776000)
necessary to get to the top of the /lib64/ld-linux-x86-64.so.2 (0x00007f7034053000)
filesystem. So, a chroot tree must

be considered merely one aspect of These object targets are usually soft
a multi-layered defense-in-depth. If LINKS REQUIRING A CHAIN OF FILES AND
your chroot tree has enough tools in links to be moved, demonstrated by
it for a cracker to gain root access, the library below:
then it’s no good; so you want to
keep the contents to the minimum # ll /lib64/libdl.so.2
necessary. In particular, don’t lrwxrwxrwx. 1 root root 13 Mar 10 2015
include any setuid-root executables! ´/lib64/libdl.so.2 -> libdl-2.17.so

# ll /lib64/libdl-2.17.so for(x in ENVIRON) print x,ENVIRON[x]
-rwxr-xr-x. 1 root root 19512 Mar 6 2015 }' > /home/jail/htdocs/awk.xyz
´/lib64/libdl-2.17.so chmod 755 /home/jail/htdocs/awk.xyz
To copy these objects and re-create If you load http://localhost/awk.xyz,

their links on Oracle Linux 7.1, follow you will see the output of the script
these steps: above. This means that, with the
added libraries, you are free to write
mkdir /home/jail/lib64 CGI scripts in GNU AWK if you wish,
cd /home/jail/lib64 even if you remove BusyBox:

cp /lib64/libdl-2.17.so . Hello, world!
ln -s libdl-2.17.so libdl.so.2
HTTP_ACCEPT text/html,application/xhtml+xml,
cp /lib64/libm-2.17.so . ápplication/xml;;q=0.9,*/*;;q=0.8
ln -s libm-2.17.so libm.so.6 AWKPATH .:/usr/share/awk
REMOTE_ADDR ::1
cp /lib64/libc-2.17.so . HTTP_ACCEPT_ENCODING gzip, deflate
ln -s libc-2.17.so libc.so.6 SERVER_PORT 80
SERVER_PROTOCOL HTTP/1.1
cp /lib64/ld-2.17.so . HTTP_ACCEPT_LANGUAGE en-US,en;;q=0.5
ln ld-2.17.so ld-linux-x86-64.so.2 CGI_PATTERN **.xyz
SCRIPT_NAME /awk.xyz
Then, copy the gawk binary and HTTP_HOST localhost
create a test script: GATEWAY_INTERFACE CGI/1.1
SERVER_SOFTWARE thttpd/2.27.0 Oct 3, 2014
cp /bin/gawk /home/jail/sbin SERVER_NAME localhost.localdomain
PATH /usr/local/bin:/usr/ucb:/bin:/usr/bin
echo '#!/sbin/gawk -f HTTP_USER_AGENT Mozilla/5.0 (X11;; Linux x86_64;;
´rv:38.0) Gecko/20100101
BEGIN { Firefox/38.0
print "Content-type: text/plain" REQUEST_METHOD GET
print ""
print "Hello, world!" GNU AWK is not the best
print "" example as it does provide network

connectivity. Brian Kernighan’s launching thttpd (you may need to

“One True AWK” might be a better install this shell):
choice at it lacks the extended
network functions. ulimit -H -f 48828
Let’s consider additional startup ulimit -H -m 48828
parameters, using systemd to control
the thttpd server. If you don’t have These commands set maximum
systemd, examine the following unit limits for memory and files written by
file and replicate it with your init thttpd and all of its child processes.
system. First, if you are still running These are specified in blocks of 1,024
thttpd, kill it: BYTES AND EQUATE TO MEGABYTES OF
maximum usage. These are hard limits
kill $(</home/jail/logs/thttpd.pid) that cannot be raised. The reason for
imposing these limits will become
Then, direct systemd to start it: clear in the next section.
The thttpd Web server records
echo "[Unit] activity with the system syslog when
Description=thttpd web service able, but when running in a chroot(),
After=syslog.target the /dev/log socket does not exist
unless created manually. The
[Service] rsyslog dæmon can be instructed
ExecStart=/bin/ksh -c 'ulimit -H -f 48828;; ulimit to listen on an additional socket in
´-H -m 48828;; /home/jail/sbin/thttpd -C /home/jail/dev/log, like so:
´/home/jail/etc/thttpd.conf'
Type=forking echo '$ModLoad imuxsock

#Restart=always $AddUnixListenSocket /home/jail/dev/log
$umask 0000' > /etc/rsyslog.d/thttpd.conf
[Install]
WantedBy=multi-user.target" > /etc/systemd/system/ mkdir /home/jail/dev
´thttpd.service chmod 755 /home/jail/dev
chcon -v --type=device_t /home/jail/dev
systemctl start thttpd.service
systemctl restart rsyslog.service
Note above the ulimit commands systemctl restart thttpd.service
executed by the Korn shell before

After restarting, you should see perspective of system resources).
thttpd entries in /var/log/messages. Assuming that you have a copy of
If you are running on an older Linux Upload-2.6.tar.gz, run the default
system that uses sysklogd, the following compile with these commands:
option will be of interest to you:
tar xvzf Upload-2.6.tar.gz
-a socket: Using this argument you cd Upload-2.6/sources/
can specify additional sockets from make
that syslogd has to listen to [sic]. ldd upload
This is needed if you’re going to let
some dæmon run within a chroot() Note that the ldd command should
environment. You can use up to 19 not be run as root on untrusted
additional sockets. If your environment software, as documented in the
needs even more, you have to manual page (run the build as a
increase the symbol MAXFUNIX regular, non-root user).
within the syslogd.c source file. The final ldd above will list the
shared object dependencies for
You also may find it useful to move the binary:
or copy the /home/jail/sbin/thttpd
binary to a location outside of the linux-vdso.so.1 => (0x00007ffcbe5e1000)
chroot(). If a copy remains in the libc.so.6 => /lib64/libc.so.6 (0x00007fbeffdad000)
jail, it can be tested at startup and /lib64/ld-linux-x86-64.so.2 (0x00007fbf00183000)
compared to the protected copy. If

the files differ, your startup script can If you previously loaded libraries
mail an alert that your jail has been above for GNU AWK, you will have all
compromised. The thttpd.conf file of the needed shared objects in place
might be similarly treated. to run this program in the chroot(). If
you have not elected to place copies of
Upload.cgi the shared objects in /home/jail/lib64,
In 2000, Jeroen C. Kessels released recompile the program with static
Upload-2.6.tar.gz, which you easily can linkage (assuming that your compiler
find using the major search engines. is capable of it—some distributions
!LTHOUGH THE SOFTWARE IS QUITE OLD IT IS lack the static libc.a):
likely the most concise implementation
of RFC 1867 available (from the gcc -static -O -o upload.static upload.c

Copy your chosen binary to curl -F file=@/etc/passwd http://localhost/test.xyz
/home/jail, and set the configuration:

Curl should return the HTML to your
cp upload /home/jail/upload.cgi standard output:
cp ../html/BadPage.html /home/jail/htdocs/test-fail.html
cp ../html/OkPage.html /home/jail/htdocs/test-good.html <html>

<body>
sed 's/action=[^ ]*/action="test.xyz"/' ../html/index.html > \ <center>
/home/jail/htdocs/test.html <h1>Success!</h1>
<hr>
cd /home/jail/htdocs
ln -s ../upload.cgi test.xyz File uploaded: passwd<br>
Bytes uploaded: 2024
echo 'Config = Default <p>
Root = /upload
FileMask = * </center>
IgnoreSubdirs = YES </body>
Overwrite = YES </html>
LogFile = /logs/upload.log
OkPage = /htdocs/test-good.html Uploaded files were configured

BadPage = /htdocs/test-fail.html to be stored in /home/jail/upload in
Debug = 0' > test.cfg this case:
If you now point your browser at # ll /home/jail/upload
http://localhost/test.html, you will total 1012
see a file upload form; test it with -rw-r--r--. 1 nobody nobody 1028368 Oct 25 10:26 foo.txt
a random file. With luck, you -rw-r--r--. 1 nobody nobody 2024 Oct 25 10:29 passwd
should see a success page, and the

file that you transferred should This configuration is powerful in
appear in /home/jail/upload. You the respect that it removes a client’s
also should see a log of the transfer ability to browse your file store if you
in /home/jail/logs/upload.log. so choose. In FTP or its descendants,
You can use the curl binary for the ability to PUT into a batch
batch transfers with this mechanism— directory also grants GET ; with this
for example: mechanism, you can constrain your

clients to transmit only, with behind upload.cgi, it is relatively
no capability to catalog or retrieve easy to extend. Consider
any content. these additional blocks for the
One potential problem with ShowOkPage() function:
this upload program is memory
consumption. Let’s examine the if (strnicmp(p1,"<insert sha256sum>",18) == 0) {
source code to upload.c: char scratch[BUFSIZ];;
FILE *H;;
/* Allocate sufficient memory for the incoming data. */
Content = (char *)malloc(InCount + 1);; *p1 = '\0';;
... sprintf(scratch,"%s%s",Root,LastFileName);;
p1 = Content;; strcpy(s1, "/sbin/sha256sum '");; strcat(s1, scratch);;
RealCount = 0;; ´strcat(s1, "'");;
/* For some reason fread() of Borland C 4.52 barfs if the
bytecount is bigger than 2.5Mb, so I have to do it if((H = popen(s1, "r")) != NULL && fgets(scratch, BUFSIZ,
like this. */ ´H) != NULL)
while (fread(p1++,1,1,stdin) == 1) { { sprintf(s1,"%s%s%s",Line,scratch,p1+18);; strcpy(Line,s1);;
RealCount++;; ´fclose(H);; }
if (RealCount >= InCount) break;; }
}
*p1 = '\0';; if (strnicmp(p1,"<insert md5sum>",15) == 0) {
char scratch[BUFSIZ];;
You can see above that the entire FILE *H;;
file is read from the network (off
standard input) and stored in memory. *p1 = '\0';;
This could be a potential “denial of sprintf(scratch,"%s%s",Root,LastFileName);;
service” exploit, thus the 50mb ulimits strcpy(s1, "/sbin/md5sum '");; strcat(s1, scratch);;
set at the end of the previous section. ´strcat(s1, "'");;
Adjust these ulimits to meet your
needs but prevent abuse. It also might if((H = popen(s1, "r")) != NULL && fgets(scratch,
be possible to use the tmpfile() ´BUFSIZ, H) != NULL)
function to spool to disk instead of { sprintf(s1,"%s%s%s",Line,scratch,p1+15);;
memory, but extensive modifications ´strcpy(Line,s1);; fclose(H);; }
TO THE # CODE WOULD BE REQUIRED }
Because there isn’t much code

Compiling in these fragments allows because the approach is flexible.

you to report the md5 and sha256 Adding custom processing on file
signatures of the on-disk data received receipt would be a nightmare for
from the client optionally, if you so FTP, but it’s relatively straightforward
specify in the template, enabling the for upload.cgi. The need for such
client to confirm that the server’s FEATURES IS ECHOED IN 3OLARIS :&3
on-disk data is correct: which performs aggressive checksums
on all disk writes—controller firmware
$ curl -F file=@Upload-2.6.tar.gz makes mistakes, and the ability to
´http://localhost/test.xyz report such errors is mandatory for
<html> some applications. Also note the
<body> signatures for Jeroen C. Kessels’
<center> package above, and further be warned
<h1>Success!</h1> that md5 signatures are vulnerable to
<hr> tampering (http://www.mathstat.dal.
ca/~selinger/md5collision)—they are
File uploaded: Upload-2.6.tar.gz<br> useful for detecting media errors, but
Bytes uploaded: 2039<br> they do not guarantee data to be free
sha256sum: bed3540744b2486ff431226eba16c487dcdbd4e60 of malicious alteration.
´2268349fdf8b7f1fb25ad38 Other useful changes to the
/upload/Upload-2.6.tar.gz upload.c code include a prefix always
<br> added to the incoming filename
md5sum: d703c20032d76611e7e88ebf20c3687a read from the .CFG file (I added
´/upload/Upload-2.6.tar.gz this feature in three lines), and
replacement of strcat() / strcpy()
<p> functions with the safer
strlcat() / strlcpy() from
</center> OpenBSD’s libc (http://www.sudo.ws/
</body> todd/papers/strlcpy.html).
</html> There is also an extended
CGI processing library
Such a data verification feature (http://www.boutell.com/cgic)
is not available in any standard in the C programming language
file transfer tool, but was easily written by Tom Boutell (author of
implemented in this elderly code the GD graphics library). The CGI-C

library offers more efficient RFC also may use a “self-signed” key that
1867 processing (without excessive will present browser errors, but will
memory consumption), but building allow encryption.
applications with it is beyond the Free signed SSL certificates should
scope of this discussion. be available by the time you read this
article from the Web site for the Let’s
Stunnel TLS Encrypt project (https://letsencrypt.org).
Because thttpd has no encryption Instructions should appear shortly
support, those living in areas on how to generate and maintain
where encryption is legal signed keys that are honored as
(http://www.cryptolaw.org) can valid by most browsers. Preliminary
use Michal Trojnara’s stunnel “shim” documentation on the Let’s Encrypt
network encryption dæmon to Web site indicates that the tools will
provide https services on port 443. use .PEM files, which likely can be
First, install the package from the used by stunnel.
standard Oracle Linux repositories: If you want to purchase a valid
key for stunnel, there is a guide
yum install stunnel on the stunnel Web site on
having your key signed by a CA
You also can install stunnel (https://www.stunnel.org/howto.html).
from source. The package pulled For more informal use, you can
BY YUM IS IN FACT QUITE OLD generate a self-signed key with the
one major version behind the following commands:
current 5.25), but it also includes
stunnel security contexts for cd /etc/pki/tls/certs
SELinux, so it is recommended make stunnel.pem
that you install the package
even if you plan to build a The process of key generation will
newer release. ASK A NUMBER OF QUESTIONS
After installation, stunnel will
REQUIRE A KEYPAIR FOR 4,3 4HE PUBLIC Generating a 2048 bit RSA private key
portion of the key can be signed by ........................................+++
a Certificate Authority (CA) if you .................................+++
wish, which will allow error-free writing new private key to '/tmp/openssl.hXP3gW'
operation with most browsers. You -----

You are about to be asked to enter information that will kQOzICzb1nt96QKdWoAob73+hv7qdi3UjJ3/20z3Cx5LWfWoa32Y50//tvBjBtcQ
be incorporated into your certificate request. H7QpiE2tfLWHTQ5tztkqVY/MZJWVgoT5LnqQlZeZB/C4izSYNo9EGAnw4ThaFJ/y
What you are about to enter is what is called a NdvmyK6sYaO3Dq4eFO78O+zzqyfhPCtcfb8lMuRTZa8uiv7ziVf0A3eGSwKYonUf
Distinguished Name or a DN. There are quite a few fields ...
but you can leave some blank -----END PRIVATE KEY-----
For some fields there will be a default value, -----BEGIN CERTIFICATE-----
If you enter '.', the field will be left blank. MIID/TCCAuWgAwIBAgIJALT/9skCvdR5MA0GCSqGSIb3DQEBCwUAMIGUMQswCQYD
----- VQQGEwJVUzELMAkGA1UECAwCSUwxEDAOBgNVBAcMB0NoaWNhZ28xGTAXBgNVBAoM
Country Name (2 letter code) [XX]:US EEFDTUUgQ29ycG9yYXRpb24xGDAWBgNVBAsMD1dpZGdldCBEaXZpc2lvbjERMA8G
State or Province Name (full name) []:IL A1UEAwwIZGFya3N0YXIxHjAcBgkqhkiG9w0BCQEWD2xpbnVzQHBvc2l4Lm9yZzAe
Locality Name (eg, city) [Default City]:Chicago Fw0xNTEwMzAwMzI2NTJaFw0yNTEwMjcwMzI2NTJaMIGUMQswCQYDVQQGEwJVUzEL
Organization Name (eg, company) MAkGA1UECAwCSUwxEDAOBgNVBAcMB0NoaWNhZ28xGTAXBgNVBAoMEEFDTUUgQ29y
´[Default Company Ltd]:ACME Corporation cG9yYXRpb24xGDAWBgNVBAsMD1dpZGdldCBEaXZpc2lvbjERMA8GA1UEAwwIZGFy
Organizational Unit Name (eg, section) []:Widget Division /JMRW5oa/+TFZIRcacTxgAw=
Common Name (eg, your name or your server's hostname) ...
´[]:darkstar -----END CERTIFICATE-----
Email Address []:linus@posix.org
4HE 02)6!4% +%9 SECTION ABOVE

The key produced above will be set is the most sensitive portion of
for expiration 365 days from the day it the file; ensure that it is not seen
was created. If you want to generate or copied by anyone you do not
a key with a longer life, you can call trust, and any recordings on backup
openssl directly: media should be encrypted. The
BEGIN CERTIFICATE section is
openssl req -new -x509 -days 3650 -nodes -out presented to TLS clients when they
´stunnel.pem -keyout stunnel.pem connect to stunnel.
It also is wise to compute custom
The key will look something like primes for the Diffie-Hellman
this (abbreviated): key exchange algorithm,
following guidance from the
# cat /etc/pki/tls/certs/stunnel.pem stunnel manual page:
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC23m+w0BLxI2zB openssl dhparam 2048 >> stunnel.pem

/p8/TiuFcEurTLbLCQwcO/FE+vNcJpddckuF6/VgpBAJk+d9i7NZNqrjMH711H18
3AYhewZTCbRUMQE3ndaYEIxSt4Qhbm8XbfUfx6Fmg4CnWh/XzE7B8Z7XbHpwRQ4d The previous command will add

another section to your stunnel.pem subject (https://hynek.me/articles/
file that looks like this: hardening-your-web-servers-
ssl-ciphers), and they represent
-----BEGIN DH PARAMETERS----- the current best practice for TLS
MIIBCAKCAQEAoHi5jzY5ZVwGCFFm1EhVsePXxNwCSs/eQbaC3rc+iXENL8xk21uq encryption. It is wise to visit this site
6eSwYIQWUeDN/h6wBBDe6dpFoNDJQeqKCmUa8aojGHnkcqsJBdVUKVF5/7rWb1Yi from time to time for news and advice
TzvbeZt8UvYnNUErJEpgBMiKPDYipE2BZ6k61WwkK6WV6svGAHpIc3o/9kU+72uf on TLS, or perhaps consider following
dPFaNIygAb2HLaJYvXq9OYGvrMsmyZTh3fnpg2RiZSVJf+i4BfyeLiYkwnSZozAS Hynek’s Twitter feed.
2rQ4hf2E5WY6jiAcNZBLKvqR8lUuIaXd9+VkiCSV0c2pXzb2ElxOk8sheAHliwip The FIPS and NO_SSL options
SaKC694z9l63eNKQW2J4WI97wkil0qa4MwIBAg== above are the default settings starting
-----END DH PARAMETERS----- with stunnel version 5. If you are
running the version 4.56 package
Once the key is in place in bundled with Oracle Linux, you must
/etc/pki/tls/certs/stunnel.pem, provide them for best practice TLS.
a stunnel configuration file must The above configuration sets
be created, like so: stunnel as an inetd-style service that
is launched for each connection. Each
echo 'FIPS = no process will be in a chroot() in
options = NO_SSLv2 /var/empty. It also is possible to run
options = NO_SSLv3 stunnel as a standing dæmon that
options = CIPHER_SERVER_PREFERENCE forks for each new client. If you do
ciphers = so, remember to restart the dæmon
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+ each time an OpenSSL update arrives
ÁES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM: and the chroot() might need more
´RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS careful preparation. If you use the
syslog = yes inetd approach, updates will apply
#debug = 6 #/var/log/secure to all new connections immediately
chroot = /var/empty after your updates, but you must use
setuid = nobody care not to exceed NPROC under high
setgid = nobody usage. There is a performance penalty
cert = /etc/pki/tls/certs/stunnel.pem for running in inetd-style, but the
connect = 127.0.0.1:80' > /etc/stunnel/https.conf ease of security administration is likely
worthwhile for all but heavy usage.
The cipher settings above are from The following commands configure
Hynek Schlawack’s Web site on the stunnel for inetd-style socket

activation under systemd: <html>
<body>
echo '[Unit] <center>
Description=https stunnel <h1>Success!</h1>
[Socket] <hr>
ListenStream=443
Accept=yes File uploaded: group<br>
[Install] Bytes uploaded: 842<br>
WantedBy=sockets.target' > /etc/systemd/system/https.socket sha256sum: 460917231dd5201d4c6cb0f959e1b49c101ea
´9ead3ab91e904eac4d758ebad4a
echo '[Unit] /upload/group
Description=https stunnel service <br>
[Service] md5sum: 31aa58285489369c8a340d47a9c8fc49
ExecStart=-/usr/bin/stunnel /etc/stunnel/https.conf ´/upload/group
StandardInput=socket' > /etc/systemd/system/https@.service
<p>
systemctl enable https.socket
systemctl start https.socket </center>
</body>
At this point, use your browser </html>
to visit https://localhost, and you

WILL SEE YOUR INDEX PAGE 6ISIT If you are using an older
https://localhost/test.html, and you Linux distribution that uses
can upload over a secure channel. xinetd, this configuration might
You also can use curl: prove useful:
curl -k -F file=@/etc/group https://localhost/test.xyz service https
{
Note above the -k option, which disable = no
disables client CA validation for a socket_type = stream
server certificate. You will need this wait = no
option if you are using a self-signed user = root
key or if your curl binary lacks access server = /usr/sbin/stunnel
to an appropriate repository of CAs server_args = /etc/stunnel/https.conf
(provided by your operating system): }

And if you are in an environment Note that if you reference files
that is still using inetd, this line will (keys or configuration files) that are
enable stunnel: not in the standard paths above, the
“enforcing” SELinux on Oracle Linux
https stream nobody nowait root /usr/sbin/stunnel 7.1 might deny read permissions. If
´stunnel /etc/stunnel/https.conf you see such errors in your syslog, try
applying the following:
If you have problems with stunnel,
try using telnet to connect to port chcon -v --type=stunnel_etc_t /alternate/path/to/https.conf
443—you may see helpful status chcon -v --type=stunnel_etc_t /alternate/path/to/stunnel.pem
messages there. For example:

If your local Linux firewall is
# cd /etc/stunnel/ enabled, you can open the port for
stunnel on https and allow remote
# mv https.conf https.conf.tmp browsers to connect. If you leave
port 80 closed, you are enforcing
# busybox-x86_64 telnet localhost 443 TLS-encrypted communication for
Clients allowed=500 all connections:
stunnel 4.56 on x86_64-redhat-linux-gnu platform
Compiled/running with OpenSSL 1.0.1e-fips 11 Feb 2013 iptables -I INPUT -p tcp --dport 443 --syn -j ACCEPT
Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP,
´FIPS Auth:LIBWRAP Please note that one drawback

Reading configuration from file to https with stunnel is that the
´/etc/stunnel/https.conf REMOTE_ADDR environment variable
Cannot read configuration shown in the above CGI scripts always
will be set to 127.0.0.1. If you want
Syntax: to determine the source of a particular
stunnel [] ] -fd | -help | -version | -sockets connection or transfer from the
- use specified config file thttpd logs, you must cross-reference
-fd - read the config file from a file descriptor them with the stunnel connection
-help - get config file help logs. However, this property
-version - display version and defaults might be useful for upload.cgi—if
-sockets - display default socket options getenv("REMOTE_ADDR")
str_stats: 1 block(s), 24 data byte(s), 58 control byte(s) != "127.0.0.1" , you should call
Connection closed by foreign host the exit() function. The net effect

is that the Web site can be visible in ´stunnel-verify.conf

clear text on both port 80 and via TLS
on port 443, but file uploads will fail The configuration above will run
if attempted in clear text. on Windows and a variety of other
Finally, if your client must ensure platforms. If you are running on a
the identity of the server, but you UNIX variant, consider also adding
do not want to obtain a signed the chroot() option in a similar
certificate, you can run a remote manner as was set on the server.
stunnel on the client that forces Note, however, that if you intend
verification on a particular key. to use the HUP signal to reload the
Extract and save the CERTIFICATE stunnel configuration, you must copy
section from your server’s stunnel ALL OF THE REQUIRED FILES INSIDE THE
.PEM (abbreviated below): chroot() to which you have confined
stunnel. Although this likely would
-----BEGIN CERTIFICATE----- never be done in an inetd-style
MIID/TCCAuWgAwIBAgIJALT/9skCvdR5MA0GCSqGSIb3DQEBCwUAMIGUMQswCQYD configuration, this is one of several
VQQGEwJVUzELMAkGA1UECAwCSUwxEDAOBgNVBAcMB0NoaWNhZ28xGTAXBgNVBAoM drawbacks for chroot() operation.
EEFDTUUgQ29ycG9yYXRpb24xGDAWBgNVBAsMD1dpZGdldCBEaXZpc2lvbjERMA8G Clients then can point their
A1UEAwwIZGFya3N0YXIxHjAcBgkqhkiG9w0BCQEWD2xpbnVzQHBvc2l4Lm9yZzAe browser at http://localhost:65432,
VYckA2gQ+70yXXxpFSD4n2ecq3ebNtej07zR2wAtAkt/JtuGiUjbl1m4ZFTPoTwr and they will be routed over TLS
... to the remote Web server. The curl
xDYMcezEgopMzYMihv6CQ0CEU+qL+92CYtEDsd1hzn74SlBK9HMKjMLrbBZPhbE4 utility similarly can use the local
/JMRW5oa/+TFZIRcacTxgAw= 65432 port in clear text, allowing
-----END CERTIFICATE----- stunnel to handle the TLS session.
When client connections are
Transfer this file to your client, and set launched, the client stunnel will
the client’s stunnel configuration file: open a connection to the server’s
port 443, then thoroughly exercise
echo 'FIPS = no the server’s key to ensure the correct
client = yes identity and prevent a “man in
verify = 4 the middle” from intercepting the
cafile = /path/to/publickey.pem transmitted data.
[client-https] The curl utility does have a
accept = 127.0.0.1:65432 number of options to import
connect = your.remote.server.com:443' > certificate stores, but it does not

appear capable of verifying a identity, integrity and privacy.
specific certificate as I have just FTP fails so completely with these
demonstrated with stunnel. QUESTIONS THAT THE ONLY JUSTIFICATION
The author of stunnel also noted for its continued use is the general
that the rsync utility and protocol familiarity with the command set. It
can be used for anonymous write is unbelievable that modern IT has
access. The standard network been thus constrained for so long.
configuration for rsync is clear text, We need a file transport protocol
but rsync also can be wrapped in with casual dropoff, integrity
either OpenSSH or stunnel. A dated checking, encryption, privilege
guide for transferring files with separation (of the TLS state machine
rsync over stunnel can be found and the HTTP file processing),
here: http://www.netbits.us/docs/ chroot() security and hooks for
stunnel_rsync.html. A benefit of custom processing on arrival. There
RFC 1867 is that curl is the only is no mainstream utility that offers
UTILITY REQUIRED FOR COMMAND all of those features.
line transfers; a more complex Until such time, RFC 1867 will
CONFIGURATION IS REQUIRED FOR AN RSYNC allow you to “roll your own”
binary to be wrapped in services transport. While this protocol has
provided by an stunnel binary. many implementations (Perl, PHP,
Python and so on), it is rare to find
Acknowledgement any with chroot() security. Hopefully,
Special thanks to Michal Trojnara, this does not remain the case. Q
the author of stunnel, for his helpful
comments on this article and his Charles Fisher has an electrical engineering degree from
greater work in stunnel development. the University of Iowa and works as a systems and database
Commercial support, licensing and administrator for a Fortune 500 mining and manufacturing
consulting for stunnel is available corporation. He has previously published both journal
from his organization. Please visit articles and technical manuals on Linux for UnixWorld and
http://www.stunnel.org/support.html other McGraw-Hill publications.
for his latest release.
Hurry Up and Downgrade Send comments or feedback via

The classic UNIX file transfer utilities http://www.linuxjournal.com/contact
ARE WOEFULLY INADEQUATE TO ENSURING or to ljeditor@linuxjournal.com.

FEATURE Transferring Conserver Logs to Elasticsearch
Transferring
Conserver Logs
to Elasticsearch
Review and search serial console logs using
Elasticsearch, Riemann and syslog-ng.
FABIEN WERNLI

I
f your organization manages Linux, Conserver
AIX, HP-UX or Solaris servers in- Conserver is a wonderful piece
house, chances are your system of software that lets you manage
administrators at least occasionally your infrastructure’s serial consoles,
need low-level access to those devices. whether they be old-style hardware
Typically, administrators use some kind of serial ports or state-of-the-art
serial console—for example, traditional Serial-over-LAN (SOL) baseboards.
serial port, Serial-over-LAN or Intelligent Its distributed design permits a
Platform Management Interface (IPMI). decentralized user experience using a
Managing and auditing console access secure, TLS-encrypted protocol. The
is not trivial, so many organizations straightforward workflow consists of
rely on the Conserver application to the user connecting to any conserver
create session logs when accessing master node, which then forwards
these servers via the serial console. the traffic to the node that manages
These logs can be useful for various the console you want to access. As all
reasons—for example, maintenance masters share the same configuration
or troubleshooting (to review why file, it is very straightforward to
something crashed), security (to find redistribute consoles among servers
out who did what—connecting user automatically (provided they are
names to actual users) or compliance virtual SOL devices, like IPMI) using
(to provide detailed session logs). configuration management (for
This article covers the following: example, using the Puppet module
we developed at CC-IN2P3).
Q How to parse and process serial So where is the catch? As far as
console logs using syslog-ng Open we are concerned at the Computing
Source Edition (Balabit). Centre of the National Institute of
Nuclear Physics and Particle Physics
Q How to send the logs to (CC-IN2P3), the logging mechanism
Elasticsearch (Elastic), so you get could be greatly improved, because
a complete, searchable audit trail CONSERVERS DESIGN IS QUITE ANCIENT
of the console access. now. For example, it does not support
logging to syslog. From a user
Q How to integrate the console logs perspective, logging is awesome, as
into a real-time monitoring system you can use a keystroke to access the
using Riemann. logs of the console, and the console

logs contain the complete session.
Useful syslog-ng From an architecture perspective

though, things are not so great, as
Features every conserver master stores the
logs of the consoles it manages locally
The syslog-ng Open Source in a file.
Edition application is a syslog
dæmon that allows you to collect syslog-ng
your log data and much more— This is where syslog-ng Open Source
for example: Edition comes into play. The idea is to
transport the logfiles of the conserver
Q Flexibly collect, parse, masters to our favorite event store
classify and correlate logs back ends, which are Riemann and
from various sources. Elasticsearch. They provide powerful
real-time stream processing and long-
Q Send log data to message
term indexed storage capabilities,
queues, including AMQP,
respectively. In addition, with syslog-ng,
STOMP or Apache Kafka.
you simply can send the logs to
Q Store your log data in plain
Riemann and Elasticsearch directly;
there is no need for any additional
files, HDFS, SQL databases
agents (like Logstash). To see how
or MongoDB.
this system works before going into
Q Forward your log data to configuration details, watch this
monitoring tools like Riemann, video: https://webcast.in2p3.fr/
Redis or Graphite. videos-syslog_ng_conserver.
The video shows what the user
Q Process CSV, JSON or does in the console (in the top-right
plain-text messages. section of the screen), its effect on
the real-time Riemann-dash dashboard
Q Rewrite, reformat and transform (bottom-right) and the near-real-time
your log messages. Elasticsearch front end (Kibana|Elastic,
on the left).
Q You can write your own modules
As you can see, the user activities
in C, Java or Python.
and events of the session are
transported to the back ends,

including useful metadata like setup is nothing special. It creates a
conserver.is_attached: true . unified logfile, which will serve as the
This tells you whether or not someone glue between conserver and syslog-ng.
was attached to the console (which is You can activate the unified logfile
obviously the case in this example). using either of the following methods:
1) Run the conserver executable with
Requirements the -U /var/log/console.log flag.
To create the system shown in 2) Use the following configuration
the demo video, you need the block in conserver.cf:
following software:
<config * { unifiedlog * /var/log/console.log;; }
Q syslog-ng Open Source Edition

3.7.2 or newer. You also can set the server’s general
logfile (where conserver stores the
Q conserver (tested with 8.2.1). global messages that are unrelated to
individual consoles)—for example, to
Q Riemann (tested with 0.2.10). /var/log/conserver.log .
Both /var/log/conserver.log
Q Elasticsearch (tested with 1.6.0). and /var/log/console.log will be
inputs for syslog-ng. You might want
Note that this article does not cover to take special care of the log rotation
how to install, configure (in general) of these files. As you are sending
and get the above software working. them to Elasticsearch, there is no need
You can find plenty of related tutorials to keep them for too long.
on-line. If you need help with these
tasks, check the documentation, Configuring syslog-ng Open
mailing lists or on-line forums for the Source Edition
software you need help with. The You need to install syslog-ng locally
following sections of this article explain on each conserver master and on a
how to configure the components central host (that is, your logserver)
of this infrastructure for the specific that will gather all the events from the
needs of our scenario as an example. conserver hosts. The local instances
will parse, process and enrich the
Configuring Conserver console output, while the central host
The conserver configuration in our will collect them and send them over

to the two back-end systems, Riemann options {
and Elasticsearch. Note that you could threaded(yes);;
get the same results using only local };;
instances, but most people prefer to
centralize first, for various reasons. source s_conserver {
Configuring syslog-ng on the channel {
Conserver Hosts: The following is an source {
example configuration file for running file(
syslog-ng on the conserver hosts. As '/var/log/conserver.log',
you can see, it has three sources: flags(no-parse)
);;
Q s_internal tracks the internal };;
messages of syslog-ng (very handy for parser {
troubleshooting, stored only locally). csv-parser(
columns(tmp.date,PROGRAM,PID,MESSAGE)
Q s_console reads the logs of the delimiters(' :')
individual consoles. quote-pairs('[]()')
flags(greedy)
Q s_conserver reads the global );;
messages of the conserver master. };;
rewrite {
The s_console and s_conserver set('$(strip $MESSAGE)', value(MESSAGE));;
sources process conserver’s unified };;
logfile. Since the format of the };;
console and conserver messages is };;
different, we have to configure
syslog-ng to parse them differently, then source s_console {
forward them to the central syslog-ng channel {
server (you can add any other sources source {
as needed for your environment): file('/var/log/console.log');;
};;
@version: 3.7 junction {
channel {
@include scl.conf filter{
program('\*/div>);;

};; port(514),
rewrite { flags(syslog-protocol)
subst('\*/div>, '', value(PROGRAM));; );;
set( };;
'true',
value('.SDATA.console.is_attached') destination d_internal {
);; file("/var/log/syslog-ng.log");;
};; };;
flags(final);;
};; log {
channel { source(s_console);;
rewrite { source(s_conserver);;
set( destination(d_remote);;
'false', };;
value('.SDATA.console.is_attached')
);; log {
};; source(s_internal);;
flags(fallback);; destination(d_internal);;
};; };;
};;
rewrite { Global Conserver Logs—the

set('$PROGRAM', value(HOST));; s_conserver Source: If you are
set('console', value(PROGRAM));; not familiar with syslog-ng, the
};; s_conserver and s_console sections
};; can be a bit intimidating. To better
};; understand how they work, take a look
at a sample message conserver produces:
source s_internal {
internal();; [Thu Sep 3 22:29:52 2015] conserver (13550):
};; ´[node42] automatic reinitialization
destination d_remote { The related source definition

network( contains three blocks:
"",
transport(tcp), 1. source : the file path and the

no-parse flag, which tells syslog-ng The first one is an unattended

to read the logfile, but not to parse message probably produced by an ACPI
it, as a syslog message (because it is signal. The second one, as hinted by
not exactly in syslog format). the * (asterisk) character appended to
the name of the console, is a message
2. parser: the csv-parser splits the produced while someone was attached
message at the colons (:) and extracts to the console (using console
the following fields: tmp.date, node66). We will use this hint to
PROGRAM, PID and MESSAGE. The produce additional metadata. The
parser’s other options ensure that the source consists again of three parts:
field values are parsed properly.
1. source: the file path, this time without
3. rewrite : defines a rewrite rule to flags. That way, syslog-ng will try to
remove leading and trailing spaces parse the message using the symbolic
from the MESSAGE key. (If you find pattern %{PROGRAM}: %{MESSAGE}.
a way to omit this point using 2., As a result, node03 and node66*
please let me know.) will be parsed into the PROGRAM key.
This configuration parses the above 2. junction: a construct with two

example message into the following mutually exclusive (hence the final
structured data: and fallback flags) channels
(similar to a “try:” “except”
tmp.date: Thu Sep 3 22:29:52 2015 structure in Python). The two
PROGRAM: conserver channels correspond to the two
PID: 13550 cases in the example: the first one
MESSAGE: [node42] automatic reinitialization for messages when someone is
attached to the console (thus, the
Console Logs—the s_console PROGRAM field contains an asterisk
Source: Here are two example character), and the second for
messages from two different consoles: messages without anybody attached.
To tell one case from the other
node03: ACPI: No handler for Region [POWR] easily (for example, when reviewing
´(ffff8808248bb150) [IPMI] the messages in Elasticsearch),
this information is stored in the
node66*: node66 login: root .SDATA.console.is_attached key.

3. rewrite : Rewrites the PROGRAM for current logs). If you absolutely
and HOST fields to their sane want to use the value from tmp.date
content: console and the name of (because, for example, you want to send
the console, respectively. old conserver logs to the remote server),
you can use the date parser from the
So in the above examples, the syslog-ng-incubator project.
messages are parsed into the Configuring syslog-ng on the
following structured data: Central Logserver: On the central
syslog-ng server, we have to route
PROGRAM: console the console and conserver messages
HOST: node03 received from the conserver hosts to the
MESSAGE: ACPI: No handler for Region [POWR] Riemann and Elasticsearch back ends.
´(ffff8808248bb150) [IPMI] The following syslog-ng configuration
.SDATA.console.is_attached: false does exactly that; the only adjustment is
that it removes the .SDATA. prefix from
PROGRAM: console the fields, so they are more readable:
HOST: node66
MESSAGE: node66 login: root @version: 3.7
.SDATA.console.is_attached: true
@include scl.conf
Forwarding the Logs to the
Central syslog-ng server, d_remote: options {
The rest of the syslog-ng configuration threaded(yes);;
is simple: we just send the structured };;
payload using the syslog IETF RFC5424
protocol (hence the syslog-protocol block destination realtime (
flag) to the central syslog-ng server. All host()
RFC5424 keys, including .SDATA.*, port(5555)
are sent over to the central syslog-ng type("udp")
server automatically. The only part that throttle(0)
we parsed from the conserver logs that flush-lines(1)
is not transferred to the central server is )
the tmp.date field. Instead, we will use {
the time when syslog-ng processes the riemann(
message (which is a good approximation flush-lines(`flush-lines`)

throttle(`throttle`) elasticsearch(
server(`host`) index("syslog-${YEAR}.${MONTH}.${DAY}"),
port(`port`) type("syslog"),
type(`type`) flush-limit(1),
ttl("${ttl:-300}") template("$(format-json -s all-nv-pairs --rekey
host("$HOST") ´.SDATA.* --shift 7 --key ISODATE)")
description("$MESSAGE") cluster("elasticsearch")
attributes( port(9300)
scope(all-nv-pairs) server("localhost")
key(".SDATA.*" client_mode("transport")
rekey( shift(7) ) time-zone("UTC")
) );;
) };;
);;
};; destination d_internal {
file("/var/log/syslog-ng.log");;
source s_remote_tcp { };;
channel {
source { destination d_riemann {
network( realtime(
transport(tcp) host("riemann"),
port(514) );;
flags(syslog-protocol) };;
tags("syslog")
so-rcvbuf(8388608) log {
);; source(s_remote_tcp);;
};; destination(d_riemann);;
};; destination(d_elasticsearch);;
};; };;

source s_internal { log {
internal();; source(s_internal);;
};; destination(d_internal);;
};;
destination d_elasticsearch {

Conclusion
From this article, you have learned
how to create a system that
Real-Time
allows you to review serial console
logs in real time and make them
Monitoring
accessible for free-text searching with Riemann
on a modern user interface. This
is helpful for maintenance and Riemann helps you monitor
troubleshooting purposes, and also distributed systems. It
for meeting auditing and compliance aggregates events from your
REQUIREMENTS 4O ACHIEVE THESE servers and applications, and
goals, conserver can be integrated it allows you to combine and
with Riemann and Elasticsearch. To process these events with a
integrate these services, you can use powerful stream processing
syslog-ng Open Source Edition, a language. You can query
flexible log collecting and processing the events and visualize
application that can collect and the results of these queries
parse the log messages and forward on dashboards. To get
them to the Riemann Elasticsearch notifications promptly, you
back end. also can trigger alerts (for
example, in e-mail or SMS).
Improvements Since the clients actively
The syslog-ng application is very push the data into Riemann,
flexible and has powerful message- your dashboards display
processing capabilities. If you learn up-to-date information (in
a bit about its possibilities, you contrast with other systems
can find several ways to improve that only pull event data every
the configuration described in the few minutes). If you use the
article. Here are some ideas that you
integrated WebSocket server,
can do with syslog-ng:
you even can have completely
synchronous event handling,
Q Write a smarter parser to
all the way from the event
extract the name of the
source to the browser.
console from server messages
(where available).

Q Correlate the console and server syslog-ng incubator project to use

messages, extract the name of the timestamp that conserver adds
the user name from the server to the messages.
messages and add them to the
console messages. That way, Fabien Wernli (faxm0dem) has been administering Linux
console events contain the clusters at the Computing Centre of the National Institute of
name of the attached user, Nuclear Physics and Particle Physics (CC-IN2P3) for 10+ years.
which makes troubleshooting Among others things, he is an expert on performance-data
and auditing easier. monitoring and infrastructure management.
Q Configure alerts for consoles that

are attached for too long. Send comments or feedback via
http://www.linuxjournal.com/contact
Q Use the date parser from the or to ljeditor@linuxjournal.com.
Linux Journal eBook Series FREE

Down
GEEK GUIDES
Practical books for the most technical people on the planet.
loa
NOW! d
Improve Finding
Business Your Way:
Processes Mapping
with an Your Network
Enterprise to Improve
Job Manageability
Scheduler Author: Bill Childers
Sponsor: InterMapper
Author: Mike Diehl
Topic: Networking
Sponsor: Skybot
Go to http://geekguide.linuxjournal.com

Resources
Demo Video: https://webcast.in2p3.fr/videos-syslog_ng_conserver
Conserver: http://www.conserver.com
Serial-over-LAN (SOL): https://en.wikipedia.org/wiki/Serial_over_LAN
Intelligent Platform Management Interface (IPMI):

https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface
Puppet Module for Conserver, Developed by CC-IN2P3:

http://github.com/ccin2p3/puppet-conserver
CC-IN2P3: http://cc.in2p3.fr
Accessing Conserver Logs with a Keystroke: http://conserver.com/docs/console.man.html
syslog-ng Open Source Edition: http://www.syslog-ng.org
Riemann: http://riemann.io
Elasticsearch: http://elastic.co/products/elasticsearch
Riemann-dash: http://riemann.io/dashboard.html
Kibana: http://elastic.co/products/kibana
syslog IETF RFC5424 Protocol: https://tools.ietf.org/html/rfc5424
Date parser from the syslog-ng-incubator project:

https://github.com/balabit/syslog-ng-incubator/tree/master/modules/date
Using correlation in syslog-ng:

https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/
syslog-ng-ose-guide-admin/html/patterndb-correlation.html
Alerting for consoles that are attached for too long:

https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/
syslog-ng-ose-guide-admin/html/patterndb-actions-correlation.html

FREE DOWNLOADS
WEBCASTS
Maximizing NoSQL Clusters for Large Data Sets
Sponsor: IBM
4HIS FOLLOW ON WEBCAST TO 2EUVEN - ,ERNERgS WELL RECEIVED AND WIDELY ACCLAIMED 'EEK 'UIDE 4AKE #ONTROL OF 'ROWING 2EDIS
.O31, 3ERVER #LUSTERS WILL EXTEND THE DISCUSSION AND GET INTO THE NUTS AND BOLTS OF OPTIMALLY MAXIMIZING YOUR .O31, CLUSTERS
WORKING WITH LARGE DATA SETS 2EUVENgS DEEP KNOWLEDGE OF DEVELOPMENT AND .O31, CLUSTERS WILL COMBINE WITH "RAD "RECHgS
INTIMATE UNDERSTANDING OF THE INTRICACIES OF )"-gS 0OWER 3YSTEMS AND LARGE DATA SETS IN A FREE WHEELING DISCUSSION THAT WILL ANSWER
ALL YOUR QUESTIONS ON THIS COMPLEX SUBJECT
> http://geekguide.linuxjournal.com/content/maximizing-nosql-clusters-large-data-sets
How to Build High-Performing IT Teams —

Including New Data on IT Performance from
Puppet Labs 2015 State of DevOps Report
Sponsor: Puppet Labs
DevOps represents a profound change from the way most IT departments have traditionally worked: from siloed teams and
HIGH ANXIETY RELEASES TO EVERYONE COLLABORATING ON UNEVENTFUL AND MORE FREQUENT RELEASES OF HIGHER QUALITY CODE )T DOESNgT
MATTER HOW LARGE OR SMALL AN ORGANIZATION IS OR EVEN WHETHER ITgS HISTORICALLY SLOW MOVING OR RISK AVERSE THERE ARE WAYS TO
adopt DevOps sanely, and get measurable results in just weeks.
> http://geekguide.linuxjournal.com/content/how-build-high-performing-it-teams-including-new-data-
it-performance-puppet-labs-2015-state
WHITE PAPERS
Comparing NoSQL Solutions
In a Real-World Scenario
Sponsor: RedisLabs | Topic: Web Development | Author: Avalon Consulting
Specializing in cloud architecture, Emind Cloud Experts is an AWS Advanced Consulting Partner and a Google Cloud
Platform Premier Partner that assists enterprises and startups in establishing secure and scalable IT operations. The
following benchmark employed a real-world use case from an Emind customer. The Emind team was tasked with
THE FOLLOWING HIGH LEVEL REQUIREMENTS
s Support a real-time voting process during massive live events

EG TELEVISED ELECTION SURVEYS OR h!MERICA 6OTESv TYPE GAME SHOWS
s +EEP VOTERS DATA ANONYMOUS BUT UNIQUE
s %NSURE SCALABILITY TO SUPPORT SURGES IN REQUESTS
> http://geekguide.linuxjournal.com/content/comparing-nosql-solutions-real-world-scenario

FREE DOWNLOADS
NEW Forrester Study!

Linux Management with Red Hat Satellite:
Measuring Business Impact and ROI
!CHIEVING !PPLICATION $ELIVERY 6ELOCITY WITH A 2/)
)"- COMMISSIONED &ORRESTER #ONSULTING TO CONDUCT ITS 4OTAL %CONOMIC )MPACT© 4%) STUDY THAT EXAMINES AND QUANTIFIES POTENTIAL
return on investment (ROI) for IBM UrbanCode Deploy within an enterprise DevOps environment. The study determined that a
COMPOSITE ORGANIZATION BASED ON THE CUSTOMERS INTERVIEWED EXPERIENCED AN 2/) OF
Read the Forrester Consulting study and learn learn how these enterprise organizations achieved:
s REDUCTION IN THE COST OF RELEASES

s Reduction in the risk of failed deployments.
s FASTER DEPLOYMENT TIMES
See how IBM UrbanCode brings deployment velocity while reducing release costs.
> http://devops.linuxjournal.com/devops/total-economic-impacttm-ibm-urbancode
Mobile to Mainframe DevOps for Dummies

In today’s era of digital disruption empowered by cloud, mobile, and analytics, it’s imperative for enterprise
organizations to drive faster innovation while ensuring the stability of core business systems. While innovative
SYSTEMS OF ENGAGEMENT DEMAND SPEED AGILITY AND EXPERIMENTATION EXISTING SYSTEMS OF RECORD REQUIRE SIMILAR
ATTRIBUTES WITH ADDITIONAL AND UNCOMPROMISING REQUIREMENTS FOR GOVERNANCE AND PREDICTABILITY )N THIS NEW BOOK
by Rosalind Radcliffe, IBM Distinguished Engineer, you will learn about:
s Responding to the challenges of variable speed IT.

s 7HY THE MAINFRAME IS A UNIQUE AND IDEAL PLATFORM FOR DEVELOPING HYBRID CLOUD APPLICATIONS
s How mobile front ends can rejuvenate back-end systems to reach new customers.
s And, special considerations for using a DevOps approach to accelerate mainframe software delivery.
> http://devops.linuxjournal.com/devops/mobile-mainframe-devops-dummies
BRAND-NEW EDITION!
DevOps For Dummies - New Edition with SAFe®
In this NEW 2nd edition, learn why DevOps is essential for any business aspiring to be lean, agile, and capable of responding
rapidly to changing customers and marketplace.
Download the E-book to learn about:
s The business need and value of DevOps.

s DevOps capabilities and adoption paths.
s How cloud accelerates DevOps.
s The Ten DevOps myths.
s And more.
> http://devops.linuxjournal.com/devops/devops-dummies-new-edition-safe

EOF
What We
DOC SEARLS
Can Do with
Ad Blocking’s
Leverage
We can do more than save publishing. We can start a
renaissance for all of business—including publishing.
We just need the code.
“Never waste a crisis”, rise in discussion of surveillance-fed

Rahm Emanuel is said to have said advertising. Here are the years when
(http://www.nytimes.com/2008/11/10/ interest in those search terms first
us/politics/10obama.html?_r=0). appeared, according to Google Trends.
And, publishers—including Linux The ones in italic and boldface are not
Journal—have one now. According to arcane to tracking-based advertising,
PageFair and Adobe, the number of but rather our response to it.
people blocking ads on their browsers
has passed 200 million, worldwide, Q 2005 — ad tag, mobile engagement.
INCREASING ANNUALLY BY IN THE
53 AND IN THE 5+ -OST OF THE Q 2006 — ad-tech, search analytics.
blockers also block tracking, which
is a main way that ads are aimed at Q 2007 — behavioral targeting,
readers through on-line publishers. retargeting, third party data,
It’s interesting to see how closely SEM tools, content analytics,
the rise in ad blocking follows the microtargeting, do not track.

EOF
Figure 1. Google Trends Graph
Q 2008 — deal id, ad fraud, social Q 2014 — supply side platform,

marketing management. data aggregators.
Q 2009 — real time message. Q 2015 — cross device tracking.
Q 2010 — demand side platform, cross- The Google Trends graph shown
device, advertising beacon, social ad in Figure 1 makes clear how people
network, predictive marketing. reacted to all this, especially after
Do Not Track failed.
Q 2011 — in-stream, real time The titles of ad blocking research
bidding, creative optimization, studies also tell a story (see Resources
search retargeting. for links). First came Ad-Blocking
Measured, published by ClarityRay
Q 2012 — clickstream data, data LATER ACQUIRED BY 9AHOO IN
management platform, mobile Then PageFair brought us The
reengagement, native advertising, Rise of Adblocking, Adblocking
adblock war. goes mainstream and The Cost of
Adblocking, in 2013, 2014 and 2015.
Q 2013 — programmatic marketing, The catch-all term for tracking-
programmatic advertising, based advertising is adtech, and
subscription push, agency trading nobody has studied or written more
desk, content marketing platform. wisely about it than Don Marti, former

EOF
Editor-in-Chief of Linux Journal. When He defines signaling this way:

I asked Don to define adtech, he said:
In a market with asymmetric
Adtech can have narrow or information, signaling is an action
broad definitions. that sends a credible message
to a potential counterparty
The narrowest definition is a system (https://en.wikipedia.org/wiki/
that implements the Fundamental Signalling_%28economics%29).
6ALUE 0ROPOSITION OF !DTECH
(as defined by Michael Tiffany, Advertising spending is a form of
http://www.whiteops.com/ signaling that shows that a seller
company)—redirecting advertising has the money to advertise (which
spending from high-value sites the seller presumably got from
to low-value sites by tracking customers, or from investors who
users. (See “Targeting Failure: thought the product was worth
legit sites lose, intermediaries win”: investing in), and believes that the
http://zgp.org/targeted- product will earn enough repeat
advertising-considered- sales to justify the ad spending.
harmful/#targeting-failure-legit-
sites-lose-intermediarieswin.) By blocking ads and tracking,
users are marking down the value of
An economic definition of digital advertising to that of spam
adtech would be any system that while also giving themselves a great
relies on information about the deal of leverage, both individually
user to reduce the signaling value and collectively.
of an advertisement. How will they use that leverage?
I see two ways: 1) encouraging
Google has search ads that have advertising with high signal
some adtech built in to them, but value; and 2) signaling their own
could work without it and probably intentions, which will be far more
better. Some other Google ad valuable than adtech’s expensive
products are pure adtech. guesswork could ever be.
The highest signal value is in old-
Facebook ads are pretty close to fashioned brand advertising. This is
pure adtech. the uncomplicated kind that sustained

EOF
all of commercial publishing and on audience members’ backs

broadcasting for centuries, never (http://www.heyrubecircus.
tracked anybody, and still makes com/fun-facts/origins-of-the-
up most of the world’s advertising carnie-lingo-mark), and put it
(including your monthly Linux to uses that people agree on and
Journal). At worst, it’s annoying don’t have to be hidden or made
and wasteful; at best, it’s useful, confusing. (Clicking “I agree” is
interesting and a form of art. (Seen not agreeing.)
any beautiful adtech lately? Or ever?)
For Madison Avenue, the mania Every set of new technologies
around adtech has marginalized the has the obvious, “hey we could
creative side of advertising, but I do THIS with it” application, and
expect that to end when the adtech surveillance marketing is the one
bubble bursts, which it will, inevitably, that a lot of people have come
given the steady growth of ad and up with for the Internet. But once
tracking blocking. Toward that, Don we can get past it—and make the
sees some synergies: Net more trustworthy for more
people—there are a lot better
IMHO, creative ad people and opportunities out there.
creative Web people could
have an awesome conference For expressing intent, we already
if we managed to exclude have intentcasting. Here the user,
all surveillance marketing from as a prospective customer, tells the
it...Web people who think market what she is ready to buy (or,
that “advertising” is creepy if she is already a customer, what
are just as mixed up as needs service).
advertising people who think I see intentcasting as a cornerstone
that “the Web” is creepy of The Intention Economy, a market
(http://zgp.org/~dmarti/business/ development I first envisioned in 2006
fresh-start/#.VkusW9DvMUU). and wrote up here in Linux Journal.
“Big Data” could be much, much 4HAT SAME YEAR ) STARTED 0ROJECT62-
more useful for everybody if at Harvard’s Berkman Center, with
people would only stop thinking the intention of making The Intention
of it as a way to automate the Economy happen. I wrote a book by
carny trick of putting chalk marks the same title in 2012, reporting on

EOF
progress thus far and forecasting what Likewise, rather than guessing
it would do for the business world. what might get the attention
Here’s the gist, from its introduction: of consumers—or what might
“drive” them like cattle—vendors
Over the coming years customers will respond to actual intentions
will be emancipated from systems of customers. Once customers’
built to control them. They will expressions of intent become
become free and independent abundant and clear, the range of
ACTORS IN THE MARKETPLACE EQUIPPED economic interplay between supply
to tell vendors what they want, how and demand will widen, and its
they want it, where and when— sum will increase. The result we
even how much they’d like to pay— will call the Intention Economy.
outside of any vendor’s system of
customer control. Customers will be This new economy will outperform
able to form and break relationships the Attention Economy that has
with vendors, on customers’ own shaped marketing and sales since
terms, and not just on the take- the dawn of advertising. Customer
it-or-leave-it terms that have been intentions, well-expressed and
pro forma since Industry won the understood, will improve marketing
Industrial Revolution.... and sales, because both will work
with better information, and
Relationships between customers both will be spared the cost and
and vendors will be voluntary and effort wasted on guesses about
genuine, with loyalty anchored what customers might want, and
in mutual respect and concern, flooding media with messages that
rather than coercion. So, rather miss their marks. Advertising will
than “targeting”, “capturing”, also improve.
hACQUIRINGv hMANAGINGv hLOCKING
in” and “owning” customers, as if The volume, variety and relevance
they were slaves or cattle, vendors of information coming from
will earn the respect of customers customers in the Intention
who are now free to bring far Economy will strip the gears
more to the market’s table than of systems built for controlling
the old vendor-based systems ever customer behavior, or for limiting
contemplated, much less allowed. CUSTOMER INPUT 4HE QUALITY OF THAT

EOF
information will also obsolete or Q GetHuman (https://gethuman.com)

re-purpose the guesswork mills of — “Need to contact a company?
marketing, fed by crumb-trails of Or have them call you? Get
data shed by customers’ mobile customer service faster and easier.”
gear and Web browsers. “Mining”
of customer data will still be useful Q Greentoe (https://www.greentoe.
to vendors, though less so than com) — “Finally...There’s a New
intention-based data provided Way to Shop! Name Your Price &
directly by customers. We Negotiate For You.”
In economic terms, there will be high Q HomeAdvisor

opportunity costs for vendors that (http://www.homeadvisor.com)
ignore useful signaling coming from — “We help you find trusted home
customers. There will also be high improvement pros.”
opportunity gains for companies that
take advantage of growing customer Q Indie Dash Button (http://www.
independence and empowerment. homeadvisor.com) — “This...
turns traditional advertising on its
.INE YEARS AFTER STARTING 0ROJECT62- head, and removes the need for
and three years after that passage was complicated targeting technology.
published, we have 23 intentcasting Customers readily identify
DEVELOPERS LISTED ON THE 0ROJECT62- themselves, creating more valuable
wiki. Here they are, with descriptions sales channels where guesswork is
from their literature: all but eliminated.” (Open source.)
Q About2Buy Q iNeed (http://www.ineedapp.com)

(https://about2buy.wordpress.com) — “Your own personal assistant.”
— “A Collaborative Commerce
System to Align Internet Buyers & Q Intently (http://intently.co) —
3ELLERS 6IA -ULTIPLE #HANNELS OF h2EQUEST ANY SERVICE ANYWHERE
Social Distribution.” with Intently.co.”
Q Crowdspending (https://www. Q Instacart (https://www.instacart.

crowdspending.com) — “...gives com) — “The best way to shop
each of us the power of all of us.” for groceries—delivered from the

EOF
stores you love in one hour.” Q RedBeacon (http://www.redbeacon.

com) — “Trusted pros for a better
Q Magic (https://www.getmagicnow. home.”
com) — “Text this phone number to
get whatever you want on demand Q TaskRabbit (https://www.taskrabbit.
with no hassle....” com) — “Tell us what you need, let
us know what we can take off your
Q Mesh (http://www.meshwithbrands. plate, choose a Tasker, hire one of
com) — “Connect with only the our fully vetted Taskers to get the
things you love....See ads from brands job done.”
that matter to you. And block the
ones that don’t.” Q Thumbtack (https://www.thumbtack.
com) — “We help you hire
Q MyTime (https://www.mytime.com) experienced professionals at a
— “Book appointments for anything.” price that’s right.”
Q MyWave (https://www.mytime.com) Q TrackIf (https://trackif.com)

— “’Frank...your very own personal — “Track your favorite sites for
assistant’, puts you in control of sales, new items, back-in-stock,
getting personalised experiences and more.”
anytime, anywhere, on any device.”
Q Webofneeds
Q Nifti (http://www.nifti.com) — (http://researchstudio-sat.github.
“One simple place to track prices io/webofneeds) — “A distributed
on the products you love.” marketplace driven by customer
needs.” (Open source.)
Q Pikaba (http://www.pikaba.com)
— “Pikaba is Social Shopping Q yellCast (http://yellcast.com) —
Platform that captures consumer “What you want, where you want it.”
intent to purchase and connects
them with the right local business.” Q :AARLY https://www.zaarly.com) —
“Hire local, hand-picked home
Q PricePatrol (http://pricepatrolapp. services. We moderate every
com) — “monitors nearby stores for job and guarantee happiness at
what you want at the price you want.” virtually any cost.”

EOF
In “Let’s scale #intentcasting”, Om.ie-like (http://customercommons.

POSTED ON THE 0ROJECT62- BLOG IN org/2013/04/25/meet-omie-a-
November, I call for “an open-source truly-personal-mobile-device)
way to scale across multiple initiative they’re calling “Rosetta
vendors with the same signaling Home”. I’m generally a critic of the
method”, while also wondering industry but http://RESO.org is an
“if there is a semantic-ish approach independent standards group and
to Intentcasting. By that I mean a their own hack (PlugFest) suggests
vernacular of abbreviated simple that their members might be open to
statements of what one is looking FutureCommerce.
for—for example: ’2br 2ba apt 10019’
means a two-bedroom and two-bath Kevin Cox of Welcomer then
apartment in the 10019 zip code.” offers this:
In the comments below, Bill Wendell
of Real Estate Café adds this to my Both vendors and customers want the
example above: Intention Economy: vendors have the
intention to sell us goods and services
Intentcasting in real estate could and customers have the intention to
deliver billions in consumer savings buy the vendor’s goods and services.
annually and open up the choke
hold on inventory (small number of Ad blocking frees us to communicate
active listings relative to demand). with whom we want. Ad blocking is
There’s also an important role better termed ad choice and ad choice
for a large scale “4th party” is part of the intention economy.
to advocate for a consumer-
centric open ecosystem as the Ad blocking is a precursor to a
industry transitions into the future change in electronic communication.
(https://blogs.law.harvard.edu/ Ad blocking, spam filters, silent
vrm/2011/04/13/fourth-parties- telephone numbers, “no junk mail”
and-vrm): signs, are what happens when we get
control over who and when and with
http://bit.ly/Back2Billions whom we communicate and connect.
If you follow @CRTLabs you’ll The Permanent Web (http://IPFS.io)

see Realtor IoT projects, and an gives both vendors and customers the

EOF
ability to create electronic permanent companies listed above) pushes

identities. Welcomer is an early back against the notion that some
permanent identity technology built generalized answer covering all of
on top of the Permanent Web. The both (as I had hoped for) might be
permanent identity from Welcomer coming: “same protocols/src is a pipe
is a set of distributed connections dream - just build already!” He’s right,
where the connections are peer to so far. (And building well.)
peer. There is no central directory. Yet lots of pipe dreams do come true.
There is no single identity. There Back in the 1980s, when I worked with
are just peer to peer connections Sun Microsystems and other competing
where each party has rights. This UNIX makers, I despaired that there
approach is likely to prevail because would ever be a free (as in freedom)
the connections are simple and *nix in the world. Too many giant tech
include all existing id systems. Putting companies were rolling their own, while
many connections of different types weirdly trying to reconcile all of them to
together leads to complexity in the !44S 362 3YSTEM 2ELEASE 4HEN
resulting emergent structures. These Linux happened. And the Web. And
are the conditions under which universal e-mail, file transfer and the rest
complexity and adaptability to fit the of it. Can we do it for intentcasting?
environment evolve. Don’t tell me. Just point to
whatever work is happening. Q
There are many ways to build a
permanent web and we can Doc Searls is Senior Editor of Linux Journal. He is also a fellow
expect there to be many variations with the Berkman Center for Internet and Society at Harvard
on IPFS.io such as CloudOS. University and the Center for Information Technology and Society
at UC Santa Barbara.
Those two comments address one
vertical market (real estate) and one
horizontal market enabler (identity). Send comments or feedback via
Meanwhile, on Twitter, James Ladd http://www.linuxjournal.com/contact
of MyWave (one of the intentcasting or to ljeditor@linuxjournal.com.
ATTENTION ADVERTISERS
The Linux Journal brand’s following has grown to a monthly readership nearly one million strong. Encompassing the
magazine, Web site, newsletters and much more, Linux Journal offers the ideal content environment to help you reach
your marketing objectives. For more information, please visit http://www.linuxjournal.com/advertising.

EOF
Resources
Ad-Blocking Measured: http://www.slideshare.net/arttoseo/clarity-ray-adblockreport
PageFair’s The Rise of Adblocking:

http://downloads.pagefair.com/reports/the_rise_of_adblocking.pdf
PageFair’s Adblocking goes mainstream:

https://blog.pagefair.com/2014/adblocking-report
PageFair’s The Cost of Adblocking: https://blog.pagefair.com/2015/ad-blocking-report
Don Marti: http://zgp.org/~dmarti
Don Marti on Signaling:

http://zgp.org/targeted-advertising-considered-harmful/#signaling
Doc Searls’ Linux Journal article “The Intention Economy” (2006):

http://www.linuxjournal.com/node/1000035
ProjectVRM: http://blogs.law.harvard.edu/vrm
Harvard’s Berkman Center: https://cyber.law.harvard.edu
The Intention Economy by Doc Searls, published in 2012:

http://www.amazon.com/The-Intention-Economy-Customers-Charge/dp/1422158527
ProjectVRM Wiki’s List of 23 Intentcasting Developers:

http://cyber.law.harvard.edu/projectvrm/VRM_Development_Work
“Let’s scale #intentcasting” by Doc Searls (November 2015):

http://blogs.law.harvard.edu/vrm/2015/11/14/lets-scale-intentcasting
Semantic Web (Wikipedia): https://en.wikipedia.org/wiki/Semantic_Web
Real Estate Café: http://realestatecafe.com
Bill Wendell’s comments on Doc’s “Let’s scale #intentcasting” post:

http://blogs.law.harvard.edu/vrm/2015/11/14/lets-scale-intentcasting/comment-page-
1/#comment-26913
Kevin Cox: http://www.welcomer.me/welcomer/?author=55ea61f8e4b05e14ae8bc98b
Welcomer: http://www.welcomer.me
Kevin Cox’s comments on Doc’s “Let’s scale #intentcasting” post:

http://blogs.law.harvard.edu/vrm/2015/11/14/lets-scale-intentcasting/comment-page-
1/#comment-26916
James Ladd: https://twitter.com/jamesladd
James Ladd’s Twitter post: https://twitter.com/jamesladd/status/665695483529072640

Instant Access to Premium
Online Drupal Training
Instant access to hundreds of hours of Drupal
training with new videos added every week!
Learn from industry experts with real world

H[SHULHQFHEXLOGLQJKLJKSURȴOHVLWHV
Learn on the go wherever you are with apps

for iOS, Android & Roku
We also offer group accounts. Give your

whole team access at a discounted rate!
Learn about our latest video releases and

RIIHUVȴUVWE\IROORZLQJXVRQ)DFHERRNDQG
7ZLWWHU#GUXSDOL]HPH
Go to http://drupalize.me and
get Drupalized today!

LinuxJournal January 2016 VK Com Stopthepress

Uploaded by

Copyright:

Available Formats

LinuxJournal January 2016 VK Com Stopthepress

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

LinuxJournal January 2016 VK Com Stopthepress

Uploaded by

Copyright:

Available Formats

™

Audit Serial WATCH:

LJ261-January2016.indd 1 12/17/15 8:35 PM

Download books for free with a

LJ261-January2016.indd 2 12/17/15 8:35 PM

Get in the Take Control

Linux in Apache Web

LJ261-January2016.indd 3 12/17/15 8:35 PM

4 / JANUARY 2016 / WWW.LINUXJOURNAL.COM

LJ261-January2016.indd 4 12/17/15 8:35 PM

86 Doc Searls’ EOF

WWW.LINUXJOURNAL.COM / JANUARY 2016 / 5

LJ261-January2016.indd 5 12/17/15 8:35 PM

President Carlie Fairchild

Publisher Mark Irgang

Associate Publisher John Grogan

Director of Digital Experience Katherine Druckman

Accountant Candy Beauchamp

Linux Journal is published by, and is a registered trade name of,

Editorial Advisory Panel

LINUX is a registered trademark of Linus Torvalds.

LJ261-January2016.indd 6 12/17/15 8:35 PM

Unsurpassed performance and reliability help build your business foundation to

Want more on cloud?

Public and Private Cloud | Managed Hosting | Dedicated Hosting | Colocation

LJ261-January2016.indd 7 12/17/15 8:35 PM

Shawn Powers runs into files without knowing exactly why

8 / JANUARY 2016 / WWW.LINUXJOURNAL.COM

LJ261-January2016.indd 8 12/17/15 8:35 PM

WWW.LINUXJOURNAL.COM / JANUARY 2016 / 9

LJ261-January2016.indd 9 12/17/15 8:35 PM

INVALID packets are those that must

10 / JANUARY 2016 / WWW.LINUXJOURNAL.COM

LJ261-January2016.indd 10 12/17/15 8:35 PM

a very similar de-jumbling algorithm word lists I found on-line). In my

WWW.LINUXJOURNAL.COM / JANUARY 2016 / 11

LJ261-January2016.indd 11 12/17/15 8:35 PM

deconstruct the input letter sets by versions, forcing me to stick to a certain

AWS EC2 VPC CLI aws ec2 describe-­instances | jq -­r

Thanks for an excellent journal. I ´'.Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId'

really enjoy it and love the digital

thing I chose to do differently in the ´select(contains({Tags: [{Key: "SomeKey",Value:

end was processing the output. I, on ´"SomeValue"} ]}) | not) | .BlockDeviceMappings[] |

occasion, had issues in processing the ´select(contains({DeviceName: "/dev/sda"})) | .Ebs.VolumeId'

sometimes changed slightly between ´-­-­description "backup_`date +\%Y\%m\%d`" -­-­volume-­id

12 / JANUARY 2016 / WWW.LINUXJOURNAL.COM

LJ261-January2016.indd 12 12/17/15 8:35 PM

parallel is a great trick to call the command on

ACCESSING THE DIGITAL ARCHIVE:

Keep up the good work! Your monthly download notifications

Photo of the Month LETTERS TO THE EDITOR: We welcome your

Argentina, already mail them to Linux Journal, PO Box 980985,

FREE e-NEWSLETTERS: Linux Journal

ljeditor@linuxjournal.com! ADVERTISING: Linux Journal is a great

WRITE LJ A LETTER and marketing opportunities by visiting

WWW.LINUXJOURNAL.COM / JANUARY 2016 / 13

LJ261-January2016.indd 13 12/17/15 8:35 PM

14 / JANUARY 2016 / WWW.LINUXJOURNAL.COM

LJ261-January2016.indd 14 12/17/15 8:35 PM

AWS EC2 VPC CLI aws ec2 describe-instances | jq -r

sometimes changed slightly between ´--description "backup_`date +\%Y\%m\%d`" --volume-id

easy: just multiply by 365.26 (to $ date --version

daysalive="$(( ( $(date -ud "$*" +'%s') - {

´$(date -u +'%s') )/60/60/24 ))" orbit=$1