Вы находитесь на странице: 1из 108

A Look at
the History
behind
systemd
Since 1994: The Original Magazine of the Linux Community MARCH 2015 | ISSUE 251 | www.linuxjournal.com

SYSTEM
ADMINISTRATION
Build Lightweight Virtual Containers
PLUS
Get a
Fully
Capable
Android
Tablet
for $20

Using Puppet’s Libreboot


Hiera and for a Free WATCH:
ISSUE
Encrypting Software OVERVIEW

Credentials Laptop
V

LJ251-March2015.indd 1 2/19/15 9:21 AM


LINUX JOURNAL
ARCHIVE DVD 1994–2014

NOW AVAILABLE
www.linuxjour nal.com/dvd
LJ251-March2015.indd 2 2/19/15 9:21 AM
Are you
you tired
tieredofof
dealing with
dealing proprietary
with storage?
proprietary storage? ®

9%2Ä4MHÆDCÄ2SNQ@FD
zStax StorCore from Silicon ZFS Unified Storage

-
From modest data storage needs to a multi-‐tiered production storage environment, zStax StorCore

zStax StorCore 64 zStax StorCore 104

The zStax StorCore 64 utilizes the latest in The zStax StorCore 104 is the flagship of the
dual-‐processor Intel® Xeon® platforms and fast zStax product line. With its highly available
SAS SSDs for caching. The zStax StorCore 64 configurations and scalable architecture, the
platform is perfect for: zStax StorCore 104 platform is ideal for:

‡VPDOOPHGLXPRIILFHILOHVHUYHUV ‡EDFNHQGVWRUDJHIRUYLUWXDOL]HGHQYLURQPHQWV
‡VWUHDPLQJYLGHRKRVWV ‡PLVVLRQFULWLFDOGDWDEDVHDSSOLFDWLRQV
‡VPDOOGDWDDUFKLYHV ‡DOZD\VDYDLODEOHDFWLYHDUFKLYHV

TalkTalk
withwith
an an
expert today:
expert today:866-‐352-‐1173
866-‐352-‐1173 -‐ http://www.siliconmechanics.com/zstax

LJ251-March2015.indd 3 2/19/15 9:21 AM


CONTENTS MARCH 2015
ISSUE 251
SYSTEM ADMINISTRATION
FEATURES
58 Using Hiera 68 Managing 82 Infinite
with Puppet Services BusyBox
Use Hiera to encrypt in Linux: with systemd
sensitive data
in Puppet. Past, Present Build one Linux
system within
Scott Lackey and Future another, using
Learn about the the latest utilities
history of init systems within the
in Linux and systemd suite of
understand how management tools.
these systems Charles Fisher
evolved over time.
Jonas Gorauskas

4 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 4 2/19/15 9:21 AM


COLUMNS 22

34 Dave Taylor’s
Work the Shell
Let’s Play Cards with Acey-Deucey,
Part II

38 Kyle Rankin’s
Hack and /
Libreboot on an X60, Part I:
the Setup

44 Shawn Powers’
The Open-Source Classroom
The Teeny Tiny $20 Tablet

100 Doc Searls’ EOF 24


Resurrecting the Armadillo

IN EVERY ISSUE
8 Current_Issue.tar.gz
10 Letters
16 UPFRONT
32 Editors’ Choice
54 New Products
105 Advertisers Index

ON THE COVER
‹(3VVRH[[OL/PZ[VY`ILOPUKZ`Z[LTKW
‹)\PSK3PNO[^LPNO[=PY[\HS*VU[HPULYZW
‹<ZPUN7\WWL[Z/PLYHHUK,UJY`W[PUN*YLKLU[PHSZW
‹3PIYLIVV[MVYH-YLL:VM[^HYL3HW[VWW
‹7S\Z!.L[H-\SS`*HWHISL(UKYVPK;HISL[MVY W
44

LINUX JOURNAL (ISSN 1075-3583) is published monthly by Belltown Media, Inc., 2121 Sage Road, Ste. 395, Houston, TX 77056 USA. Subscription rate is $29.50/year. Subscriptions start with the next issue.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 5

LJ251-March2015.indd 5 2/19/15 11:53 AM


Executive Editor Jill Franklin
jill@linuxjournal.com
Senior Editor Doc Searls
doc@linuxjournal.com
Associate Editor Shawn Powers
shawn@linuxjournal.com
Art Director Garrick Antikajian
garrick@linuxjournal.com
Products Editor James Gray
newproducts@linuxjournal.com
Editor Emeritus Don Marti
dmarti@linuxjournal.com
Technical Editor Michael Baxter
mab@cruzio.com
Senior Columnist Reuven Lerner
reuven@lerner.co.il
Security Editor Mick Bauer
mick@visi.com
Hack Editor Kyle Rankin
lj@greenfly.net
Virtual Editor Bill Childers
bill.childers@linuxjournal.com

Contributing Editors
)BRAHIM (ADDAD s 2OBERT ,OVE s :ACK "ROWN s $AVE 0HILLIPS s -ARCO &IORETTI s ,UDOVIC -ARCOTTE
0AUL "ARRY s 0AUL -C+ENNEY s $AVE 4AYLOR s $IRK %LMENDORF s *USTIN 2YAN s !DAM -ONSEN

President Carlie Fairchild


publisher@linuxjournal.com

Publisher Mark Irgang


mark@linuxjournal.com

Associate Publisher John Grogan


john@linuxjournal.com

Director of Digital Experience Katherine Druckman


webmistress@linuxjournal.com

Accountant Candy Beauchamp


acct@linuxjournal.com

Linux Journal is published by, and is a registered trade name of,


Belltown Media, Inc.
PO Box 980985, Houston, TX 77098 USA

Editorial Advisory Panel


Nick Baronian
Kalyana Krishna Chadalavada
"RIAN #ONNER s +EIR $AVIS
-ICHAEL %AGER s 6ICTOR 'REGORIO
$AVID ! ,ANE s 3TEVE -ARQUEZ
$AVE -C!LLISTER s 4HOMAS 1UINLAN
Chris D. Stark

Advertising
% -!),: ads@linuxjournal.com
URL: www.linuxjournal.com/advertising
0(/.%     EXT 

Subscriptions
% -!),: subs@linuxjournal.com
URL: www.linuxjournal.com/subscribe
MAIL: PO Box 980985, Houston, TX 77098 USA

LINUX is a registered trademark of Linus Torvalds.

LJ251-March2015.indd 6 2/19/15 9:21 AM


LJ251-March2015.indd 7 2/19/15 9:21 AM
Current_Issue.tar.gz

Putting Out SHAWN POWERS

Fires and
Designing Fire-
Proof Buildings
S
ystem administration is a very and ever-changing field. This
general term. It’s our job to month, we learn how to be better
fix problems, repair systems at our jobs, even if the measure of
and remind people to try power “success” is constantly fluctuating.
cycling their troubled desktops. Dave Taylor starts off this issue with
We are also responsible for a continuation of his script-based
creating systems that don’t develop card game. Designing games with
problems, need fewer repairs and Dave is a great way to become better
run without being power cycled. In shell scripters, and so in a very real
an ideal world, system administrators sense, we can justify playing games
would work themselves out of a at work. Kyle Rankin follows Dave
job in short order. Thankfully (or with a nerdier sort of game: trying
unfortunately?), that’s not how it to replace the proprietary BIOS on
goes. We always have problems a ThinkPad with Libreboot. Coreboot
to fix, and there’s always a better is an open-source BIOS replacement,
way to do what we’re doing. Thus, and Libreboot goes a step further
system administration is a vibrant by stripping out all the proprietary
code. If you think having a free
VIDEO: BIOS with built-in GRUB sounds
V

Shawn Powers runs interesting, you’ll want to check out


through the latest issue.
Kyle’s column this month.

8 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 8 2/19/15 9:21 AM


CURRENT_ISSUE.TAR.GZ

My personal contribution to the article. Once you understand systemd,


System Administration issue is Charles Fisher follows up with a great
something I find to be more useful tutorial on using the new init system
than I ever expected. Android tablets to create powerful and lightweight
are convenient for things like Wi-Fi virtual containers utilizing systemd
sniffing, but they are often unwieldy FOR INITIALIZATION &OR STUBBORN 3YS6
to carry around. My solution is to lovers like myself, it’s great to read
convert a cheap pre-paid cell phone some information on the advantages
into a tiny, pocket-size tablet. If you systemd might offer.
already have an Android phone, it Doc Searls closes out our issue with
might be redundant, but for me, a a new look at the 15-year-old Cluetrain
$20 tablet was too hard to pass up. In Manifesto. If you’re a fan of the Locke,
my column, I give you all the details. Levine, Weinberger and Searls project,
Puppet is an incredible tool for you’ll want to read what’s happening
managing the system configurations of with New Clues today.
multiple nodes. Scott Lackey describes If it weren’t for the modern
a great tool we can use to store site- technological world we live in, system
specific data more efficiently (and administration wouldn’t even exist!
securely). Hiera is a key/value lookup Thankfully (or again, unfortunately?),
tool that integrates directly with our world is getting more and more
Puppet and makes a great tool even technological every day. The need
better. If you want to have a clear for system administrators and their
separation between your sensitive data tools are more in demand than ever
and the Puppet system that uses it, before, and this issue of Linux Journal
or if you want to save time by reusing was written to educate, inform and
common data, Hiera is a tool any even entertain those of us in the
Puppet admin will want to check out. digital trenches. We hope you enjoy
Jonas Gorauskas gives us a history this issue as much as we enjoyed
of systemd. Whether you love the putting it together! Q
new initialization system, or think
it’s a terrible implementation of a Shawn Powers is the Associate Editor for Linux Journal .
horrible idea, systemd is here to He’s also the Gadget Guy for LinuxJournal.com, and he has
stay—at least for a while. If you’ve an interesting collection of vintage Garfield coffee mugs.
ever been curious how we got Don’t let his silly hairdo fool you, he’s a pretty ordinary guy
FROM SIMPLE INIT SCRIPTS TO 3YS6 AND and can be reached via e-mail at shawn@linuxjournal.com.
beyond, you’ll want to read Jonas’ Or, swing by the #linuxjournal IRC channel on Freenode.net.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 9

LJ251-March2015.indd 9 2/19/15 9:21 AM


letters
I’m glad you’re back in the fold,
welcome home!—Shawn Powers

Vagrant Simplified
2EGARDING 3HAWN 0OWERS h6AGRANT
Simplified” in the January 2015
ISSUE GREAT ARTICLE ) TRIED 6AGRANT
a few months back, and I couldn’t
get the light bulb to turn on. Thus,
I put it aside. Shawn’s article
supplied the understanding I was
missing. Many thanks.
—Tim Parks

Digital Format That’s exactly what I was hoping


An interesting thing happened. I for! I’m glad it worked, and I’m glad
dropped reading LJ a while back due Vagrant is demystified for a few
to hating to stare into a monitor. more people. Thank you for the kind
But last week I finally purchased an words.—Shawn Powers
Amazon tablet and re-subscribed to LJ
because of the LJ app. It’s now easy Suggestion for Dave Taylor
TO READ ON A NICE SCREEN %VEN THOUGH ! WHILE AGO ) WROTE A # PROGRAM
I still enjoy printed magazines, I do for downloading stock and option
respect the environment and agree information from Yahoo in Windows.
that chopping down green for this I remember it took a lot of code
is not good. So good choice on an to parse the information, most
environmentally-friendly mag. particularly the option information.
—Peter K.
Since then, I have graduated to Linux,
Thanks Peter! Paper magazines and I am currently running on Xubuntu.
have a dear place in my heart as ! FRIEND PIQUED MY INTEREST IN OPTION
well, but I can’t deny the digital trading that caused me to revisit coding
format has some advantages too. a Linux version of option tracking. I have

10 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 10 2/19/15 9:21 AM


[ LETTERS ]

not yet gotten around to coding any GUI 0.00


STUFF AND ) WANTED SOMETHING QUICKLY SO 10
) JUST WROTE A FEW # PROGRAMS USING 21
Geany to run on the terminal. 65.14

I really didn’t want to write hundreds Of course, with a slight change to:
of lines of code to parse the data and
looked around for some XML parsers. wget -q -O /tmp/_option.html

None looked easy enough for me to ´http://finance.yahoo.com/q/op?s=SPY&date=1429228800

use, but then I looked at the source ´&& sleep 1 && grep

page, which I downloaded, and tried ´'option_entry\|:volume' /tmp/_option.html | sed -n

grep , which led me to develop the ´'s/\r//;s/[^>]*//;s/>//;s/<\/div>//;p' | sed

following few, or one, line(s) of code ´'s/<\/strong>//;s/[^>]*>//;s/<\/a>//;s/%//'

that I thought you might be interested ´| grep -A8 SPY141226P00230000

as a source for some future articles:


I could say that we can use “one line”
wget -O /tmp/_option.html of code to parse the Yahoo finance
´http://finance.yahoo.com/q/op?s=SPY&date=1429228800 page for the option information!

grep 'option_entry\|:volume' /tmp/_option.html | sed -n 4HIS IS # IN THAT ) USE SYSTEM TO


´'s/\r//;s/[^>]*//;s/>//;s/<\/div>//;p' | sed execute. Maybe a popen function
´'s/<\/strong>//;s/[^>]*>//;s/<\/a>//;s/%//' > /tmp/_option.txt might be better as an alternative, but
I didn’t think of it at the time.
grep -A8 SPY141226P00230000 _option.txt

In summary, I thought this was pretty


These three lines extracted the cool, and you may have already done
following information: something similar, but as I said, I
thought it might give you some ideas
SPY141226P00230000 for future articles. I enjoy and find
24.27 your articles educational, which are
22.87 usually one of the first I read after the
24.09 titles that catch my eye.
0.00 —Roger

WWW.LINUXJOURNAL.COM / MARCH 2015 / 11

LJ251-March2015.indd 11 2/19/15 9:21 AM


[ LETTERS ]

Dave Taylor replies: Thanks for against malicious file corruption—


your note and code snippet, Roger. for example, an administrator
It is rather amazing what you can do inserting random bits into the files.
with sed, although when it gets that
complex, you might consider having the Also, for readers who are interested in
script in a separate file and using the using zbackup to back up very large
-f FILE option to sed to retain your directory structures, there is a pre-release
sanity as you debug it. The problem of software on https://github.com/
with all of these crude HTML parsers, of davidbartonau/zbackup-tar that
course, is that if they make the slightest backs up directories about 10x faster
tweak on the page, your code’s broken. on a non-SSD.
I know; it happens to me all the time. —David Barton

Response to Letter in the January Kyle Rankin’s Dr Hjkl on the


2015 Issue Regarding zbackup Command Line
Regarding Chris W ills’ letter in Regarding Kyle Rankin’s article “Dr
the January 2015 issue [this letter Hjkl on the Command Line” in the
is from David Barton, author of $ECEMBER  ISSUE OF LJ: it seems
the article “Ideal Backups with that Mr Rankin wants to use vi
ZBACKUPv IN THE .OVEMBER  keystrokes to manipulate the shell
issue]: currently I use rsnapshot to command line, so why is he explaining
back up the zbackup stores with %MACS MODE KEYSTROKES )N THE SHELL
hourly, daily, weekly rotations. all he needs to type is set -o vi
Because the zbackup store changes and use vi mode from then on. Hit
very slightly each time, it is very %SC TO ENTER COMMAND MODE THEN
space-effective. Due to the IO load hop to the previous word with b, next
caused by large numbers of files, word with w, change the current word
very large numbers of servers may WITH CW AND SO ON %VEN hHJKLv ARE
want to look at options that don’t active, for moving the cursor to the
REQUIRE LINKING ALL THE FILES SUCH AS previous/next letter/shell command.
rotating thin provisioned snapshots,
"TRFS:&3 SNAPSHOTS OR ROTATING ONTO The mechanism is called GNU
removable storage media like tape. READLINE IT SUPPORTS BOTH %MACS
I don’t think the snapshots need (default) and vi mode (put set
to be replicated, since it is a guard editing-mode vi in ~/.inputrc),

12 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 12 2/19/15 9:21 AM


[ LETTERS ]

and most command-line tools like the with the vi command set, as I am too.
SHELL OR THE -Y31, CLIENT OR THE '$" He then goes on to describe bash
debugger will behave accordingly command-line editing capabilities,
because they’re using the library. using lots of Ctrl and Alt keys. If you’ve
—Mike ever been sucked into editor wars, I’m
sure it occurred to you that those key
Dr Hjkl on the Command Line, II SEQUENCES SEEM AWFULLY %MACS LIKE
)N THE $ECEMBER  ISSUE SERIOUSLY
Kyle: set -o vi . And, in fact, that’s exactly what they are.
—Xaveer Bash starts out with its command-line
EDITING IN %MACS MODE (OWEVER BASH
Dr Hjkl on the Command Line, III also has a perfectly functional vi mode
)N +YLE 2ANKINS $ECEMBER  that may seem more familiar to you. Just
column, he describes being comfortable do set -o vi to turn on vi mode.

LJ251-March2015.indd 13 2/19/15 9:21 AM


[ LETTERS ]

Now, having said that, I have to time. W ithout the digital format,
admit that I leave my bash sessions it’s hard to keep reading such a
IN %MACS MODE ALMOST EXCLUSIVELY fine magazine. Seriously, I do not
The vi mode, like the vi editor, is understand 60% of what is written,
modal, and that modality is somewhat but if one keeps reading, surely
non-intuitive in command-line one’s knowledge will gradually
editing. However, for a vi fan, it’s improve. Keep up the good work.
certainly worth exploring. —KokYY
—Tim Roberts
Awesome! Yes, please keep reading.
Kyle Rankin replies: I remember Then after a couple months go
when I first got really interested back and see if any of the older
in vi that I changed the command stuff makes sense. (Don’t worry if
line to vi mode. I realized pretty it doesn’t all make sense, however;
quickly though that I didn’t like sometimes the articles make my
having modes on the command head spin too!)—Shawn Powers
line and switched it back.
In general, I try to keep my They Said It
environments set to their defaults, I just wanted to put in a good
so you won’t find me with custom word for the They Said It column.
bashrc files that set a lot of aliases 4HE QUOTES ARE NOT ALWAYS
or anything like that. It’s just too memorable (although they often
much of a pain to ship custom are), but they always put me in a
settings like that throughout all good frame of mind for enjoying
my home and work systems, so the rest of the issue.
instead, I try to make the most —Steven Janke
with the defaults I get.
Thank you Steven. I enjoy looking
Digital Format for good quotes every month. The
Happy New Year to you and your hardest part is making sure I don’t
team. I had stop subscribing to this repeat any (unless they’re really
magazine some time ago and came good ones!)—Shawn Powers
back because of the digital format.
Why? Because I am a seaman Kyle Rankin’s EC2 Security Groups
who is away for six months at a In the “Secure Server Deployments

14 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 14 2/19/15 9:21 AM


in Hostile Territory” article in the January
2015 issue, Mr Rankin says: “... it’s important
to know that Security Groups are assigned
At Your Service
only when an instance is created—you can’t SUBSCRIPTIONS: Linux Journal is available
in a variety of digital formats, including PDF,
add or remove Security Groups from an .epub, .mobi and an on-line digital edition,
as well as apps for iOS and Android devices.
instance after you create it.” Renewing your subscription, changing your
e-mail address for issue delivery, paying your
invoice, viewing your account details or other

)N A 60# WHICH HAS BEEN %#S DEFAULT FOR A subscription inquiries can be done instantly
on-line: http://www.linuxjournal.com/subs.
good while now, this is not true. You easily E-mail us at subs@linuxjournal.com or reach
us via postal mail at Linux Journal, PO Box
can change the Security Group(s) associated 980985, Houston, TX 77098 USA. Please
remember to include your complete name
with an instance. and address when contacting us.

—Gx ACCESSING THE DIGITAL ARCHIVE:


Your monthly download notifications
will have links to the various formats
Kyle Rankin replies: Thanks for the e-mail. and to the digital archive. To access the
digital archive at any time, log in at
It could be that I’m just showing my age in http://www.linuxjournal.com/digital.

how long I’ve been working with EC2, since LETTERS TO THE EDITOR: We welcome your
letters and encourage you to submit them
none of my accounts default into VPCs, and at http://www.linuxjournal.com/contact or

I’m still in the wild west of “EC2 Classic”. mail them to Linux Journal, PO Box 980985,
Houston, TX 77098 USA. Letters may be
That said, I do think having to think in terms edited for space and clarity.

of the limitations of the classic EC2 Security WRITING FOR US: We always are looking
for contributed articles, tutorials and
Group model helps build more robust security real-world stories for the magazine.
An author’s guide, a list of topics and
since you can take less for granted. due dates can be found on-line:
http://www.linuxjournal.com/author.

FREE e-NEWSLETTERS: Linux Journal


editors publish newsletters on both
a weekly and monthly basis. Receive
late-breaking news, technical tips and
tricks, an inside look at upcoming issues
and links to in-depth stories featured on

WRITE LJ A LETTER http://www.linuxjournal.com. Subscribe


for free today: http://www.linuxjournal.com/
We love hearing from our readers. Please enewsletters.

send us your comments and feedback via ADVERTISING: Linux Journal is a great
resource for readers and advertisers alike.
http://www.linuxjournal.com/contact. Request a media kit, view our current
editorial calendar and advertising due dates,
or learn more about other advertising
and marketing opportunities by visiting
PHOTO OF THE MONTH us on-line: http://ww.linuxjournal.com/
advertising. Contact us directly for further
Remember, send your Linux-related photos to information: ads@linuxjournal.com or
ljeditor@linuxjournal.com! +1 713-344-1956 ext. 2.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 15

LJ251-March2015.indd 15 2/19/15 9:21 AM


UPFRONT NEWS + FUN

diff -u
WHAT’S NEW IN KERNEL DEVELOPMENT
Nicolas Dichtel and Thierry But, he felt that the whole linked
Herbelot pointed out that the list concept was not the right
directories in the /proc filesystem APPROACH %SPECIALLY HE FELT THAT
used a linked list to identify their /proc/net/dev/snmp6 was the real
files. But, this would be slow when target of Nicolas and Thierry’s
/proc directories started having lots patch, and if no one actually
of files, which, for example, might needed the files in that directory
happen when the system needed EXCEPT PEOPLE REQUIRING EXTREME
lots of network sockets. backward compatibility), it would
Nicolas and Thierry posted be even more efficient to do away
a patch to change the /proc with them completely.
implementation to use multiple This, however, already had come
linked lists instead of just one. up in an earlier thread, when
%ACH SUBDIRECTORY WOULD HAVE ITS David S. Miller had said that
own linked list, keyed to a hash of “It potentially breaks tools, it’s a
the directory’s name. According to non-starter, sorry.” So, reworking
their benchmarks, the patch shaved the user interface would not be
1/5 of the time needed to chur n allowed, which left the linked list
through all the entries of a given speedup that Nicolas and Thierry
subdirectory. proposed. But, Nicolas said he’d
Stephen Hemminger liked the look into an rbtree implementation
speedup, but suggested that there instead of a plain linked list,
already were implementations, because rbtrees would potentially
like the hlist macro, that might scale better.
simplify their hash table code. Minchan Kim noticed that
Eric W. Biederman also liked putting memory pressure on
the speedup and kicked himself qemu-kvm UNDER ,INUX  WOULD
for overlooking the /proc issue cause a ker nel stack overflow and
when doing other scalability work. crash the system. He dug into the

16 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 16 2/19/15 9:21 AM


[ UPFRONT ]

code and tried to reduce his own ON X  BUT NOBODY OUTSIDE
stack usage, but he wasn’t able filesystem and IO developers
to cut back enough to prevent the has been willing to accept that
crash. And in any case, he said, argument as valid, despite
trying to reduce everyone’s stack regular stack overruns and
usage was not very scalable. He filesystems having to add
proposed expanding the ker nel workaround after workaround
stack from 8K to 16K, although he to prevent stack overruns.”
acknowledged that there possibly He added, “We’re basically at
were good reasons not to do this the point where we have to push
that he wasn’t aware of. EVERY 8&3 OPERATION THAT REQUIRES
Dave Chinner remarked that block allocation off to another
“8k stacks were never large enough thread to get enough stack space
to fit the Linux IO architecture for normal operation”, and said

LINUX JOURNAL
now available
for the iPad and
iPhone at the
App Store.

linuxjournal.com/ios
For more information about advertising opportunities within Linux Journal iPhone, iPad and
Android apps, contact John Grogan at +1-713-344-1956 x2 or ads@linuxjournal.com.

LJ251-March2015.indd 17 2/19/15 9:21 AM


[ UPFRONT ]

“XFS has always been the stack version. Linus also pointed out that
usage canary and this issue is there was plenty of room to reduce
BASICALLY A REPEAT OF THE K STACK stack usage in the stack trace
on i386 ker nel debacle.” Minchan had posted in his original
Borislav Petkov pointed out e-mail. Linus remarked, “From a
that if they increased the ker nel QUICK GLANCE AT THE FRAME USAGE
stack from 8K to 16K, there some of it seems to be gcc being
undoubtedly would come a time rather bad at stack allocation,
when 16K wouldn’t be enough but lots of it is just nasty spilling
either. He wondered if there ever around the disgusting call-sites
would be a limit, or if the ker nel with tons or arguments. A lot
stack ultimately would grow to of the stack slots are marked as
one megabyte and beyond. ’%sfp’ (which is gcc-ese for ’spill
Steven Rostedt said, “If frame pointer’, afaik).”
[Minchan’s patch] goes in, it There was a technical discussion
should be a config option, about various ways to reduce stack
or perhaps selected by those usage in general (and some further
filesystems that need it. I hate consideration of ways in which
to have 16K stacks on a box that GCC might be somewhat to blame),
doesn’t have that much memory, but with Linus willing to accept
but also just uses ext2.” a patch to implement a larger
Meanwhile, H. Peter Anvin said, stack, it seems like something
“8K additional per thread is a huge along the lines of Minchan’s patch
hit. XFS has indeed always been a will soon be part of the ker nel.
canary, or trouble spot, I suspect At one point, Linus summed up
because it originally came from his position on the issue, saying,
another ker nel where this was not “Minchan’s call trace and this
an optimization target.” thread has actually convinced me
At around this point, Linus that yes, we really do need to make
Torvalds remarked that something X  HAVE A K" STACK ;=
like Minchan’s fix probably would The 8kB stack has been somewhat
be necessary at some point, restrictive and painful for a while,
although the development cycle and I’m ok with admitting that it is
was already at -rc7, making it just getting too damn painful.”
too late for that particular ker nel —ZACK BROWN

18 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 18 2/19/15 9:21 AM


March 16–19, 2015 | Boston, MA

From strategies to essential


technologies—build a solid
foundation in software architecture
The O’Reilly Software Architecture Conference is a new event
designed to provide the in-depth professional training that
software architects and people working on software architecture
need to support the success of their businesses.
■ Reactive and its variants ■ Continuous Deployment
■ Microservices ■ Architecture Fundamentals
■ Continuous Delivery ■ Business Skills
■ Integration Architecture
■ Devops
■ Scaling
■ Big Data

Save 20%
softwarearchitecturecon.com
on your ticket
@oreillysacon Use code LINUXJ

LJ251-March2015.indd 19 2/19/15 9:21 AM


[ UPFRONT ]

Android Candy: They Said It

Bluetooth Auto Do something every


day that you don’t

Connect want to do; this is


the golden rule for
acquiring the habit
I love my latest Android device (see this issue’s of doing your duty
Open-Source Classroom column for details), but without pain.
for some reason, it won’t automatically connect —Mark Twain
to my Bluetooth headset. When I turn on my
headset, I want it to connect to my Android It’s okay if you mess
device so I can start using it right away. In order up. You should give
yourself a break.
to make it connect, I have to go into the settings
—Billy Joel
app, then Bluetooth, and then tap the device to
connect. Thankfully, there’s an application that Let me tell you the
makes life a lot easier. secret that has led
Bluetooth Auto Connect is a program that runs in me to my goal. My
the background. It doesn’t constantly poll for newly strength lies solely
turned on Bluetooth devices, because that would in my tenacity.
waste battery power. It has several other ways to —Louis Pasteur
initiate the connection though. My favorite is the
“connect when powered on” option. Because I If you limit your
always have to turn the phone on in order to start choices only to what
my audiobook (or music), it’s not an inconvenience seems possible or
reasonable, you
to turn the screen on in order to connect Bluetooth.
disconnect yourself
As soon as the power button is pressed, it connects
from what you truly
to my headset, and by the time I open the media
want, and all that is
player application, it’s ready to rock!
left is a compromise.
Sometimes it’s the simplest applications that are
—Robert Fritz
the most useful. Bluetooth Auto Connect is one
of those. Check it out in the Google Play Store The highest result
today: https://play.google.com/store/apps/ of education is
details?id=org.myklos.btautoconnect. tolerance.
—SHAWN POWERS —Helen Keller

20 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 20 2/19/15 9:21 AM


LJ251-March2015.indd 21 2/19/15 9:21 AM
[ UPFRONT ]

Non-Linux FOSS:
MenuMeters
It sounds like a “back in my day” story, LAPTOPS HAD ,%$ ACTIVITY LIGHTS FOR HARD
but I really do miss the days when drives and Wi-Fi. Sure, some still have

Menu Bar (screenshot from http://ragingmenace.com)

Customizing MenuMeters

22 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 22 2/19/15 9:21 AM


[ UPFRONT ]

them, but for the most part, the latest MenuMeters application. It puts all sorts
trend is to have no way of knowing if of monitoring ability right in your menu
your application is pegging the CPU at bar. MenuMeters supports CPU activity,
100%, or if it just locked up. network activity and even memory
The hardware on Apple-branded usage. With a wide range of display
LAPTOPS IS AMAZING %VEN IF YOU HATE options, you can customize MenuMeters
the operating system, the solid to be as informative or subtle as you like.
aluminum cases are just awesome. MenuMeters is licensed under
Like most other brands of laptops, the GPL and is available to download
however, they lack any activity lights. A at http://www.ragingmenace.com.
perfect fix for OS X is the open-source —SHAWN POWERS

Tighten Up SSH
SSH is a Swiss Army knife and Hogwart’s available as well. Disabling the old
magic wand all rolled into one simple SSH version 1 protocol is as simple as
command-line tool. As often as we use it, we changing (or adding):
sometimes forget that even our encrypted
friend can be secured more than it is by Protocol 2, 1
default. For a full list of options to turn on
and off, simply type man sshd_config to Change it to:
read the man page for the configuration file.
As an example, one of the first things Protocol 2
I do is disable root login via SSH. If you
open /etc/ssh/sshd_config as root, search Then only the far more secure version
for a line mentioning PermitRootLogin  PROTOCOL WILL BE ABLE TO CONNECT %VERY
and change it to no . If you can’t find a server situation has different security
line with that option, just add it to the needs. Reading through the man page
end. It will end up looking like: might reveal some options you never even
considered before. (Note that the sshd
PermitRootLogin no dæmon will need to be restarted for the
changes to be applied. Or, if in doubt, just
Plenty of other security options are reboot the computer.) —SHAWN POWERS

WWW.LINUXJOURNAL.COM / MARCH 2015 / 23

LJ251-March2015.indd 23 2/19/15 9:21 AM


[ UPFRONT ]

Solving ODEs on Linux


Many problems in science and handle these dependencies yourself.
engineering are modeled through Included with the source is a
ORDINARY DIFFERENTIAL EQUATIONS /$%S directory of examples. You can use
http://en.wikipedia.org/wiki/ them as a starting point and to gain
Ordinary_differential_equation). some ideas of what you can do with
!N /$% IS AN EQUATION THAT CONTAINS A Model Builder. Documentation is a
function of one independent variable bit sparse, so you may need to get
and its derivatives. This means that your hands a little dirty to take the
practically any system that changes most advantage of what is possible
over time can be modeled with an with Model Builder.
/$% FROM CELESTIAL MECHANICS TO To start Model Builder, you either
chemistry reaction rates to ecology can click on its menu item in your
and population modeling. desktop environment or run the
"ECAUSE OF THIS UBIQUITY MANY TOOLS command PyMB from a terminal
have been developed through the years window. When the main window
TO HELP SOLVE AND ANALYZE /$%S )N pops up, you are presented with
this article, I take a look at one of the a template where you can define
tools available on Linux: Model Builder the problem you are analyzing
(http://model-builder.sourceforge.net). (Figure 1). The main pane, titled
The project is hosted on SourceForge, $IFFERENTIAL %QUATIONS IS WHERE
so you always can build it from you can define the set of ordinary
source, but most distributions should DIFFERENTIAL EQUATIONS THAT YOU ARE
have a package available. On Debian- trying to solve. The general form of
based distros, you can install THESE EQUATIONS IS DYDT  FY T 
it with the command: If your system depends on
different levels of differentiating
sudo apt-get install model-builder the dependent variable, you always
CAN REWRITE IT AS A SYSTEM OF /$%S
It also installs several Python When you give Model Builder your
modules to support the tasks it system, you need to write out
can handle. If you do decide to only the right-hand side of the
build from source, you will need to ABOVE EQUATION 4HIS EQUATION CAN

24 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 24 2/19/15 9:21 AM


[ UPFRONT ]

Figure 1. When Model Builder starts, you can set several parameters and the equations
you want to analyze.

contain essentially any function or The pane to the right of the


expression that NumPy understands, EQUATION WINDOW IS WHERE YOU CAN
since Model Builder uses Python to place any parameters that you need,
do the heavy lifting. one per line. They can be used in
Because Model Builder is designed THE EQUATION WINDOW WHERE THEY
TO HANDLE SYSTEMS OF EQUATIONS are labeled as p[0], p[1] and so on.
you need to define the y portion as If you want to use time in either the
elements of a list. So the y variable PARAMETERS OR EQUATIONS THAT YOU
FOR THE FIRST EQUATION IS LABELED AS have defined, you just need to use
y[0]; the y variable for the second the t variable.
EQUATION IS LABELED Y;= AND SO ON Because Python is used in the
These are called the state variables. back end, you even can use lambda

WWW.LINUXJOURNAL.COM / MARCH 2015 / 25

LJ251-March2015.indd 25 2/19/15 9:21 AM


[ UPFRONT ]

functions to define more complex values for each state variable at


structures. You may want to take THE TIME T 4HEY NEED TO BE
a look at the documentation separated with a space and put in
available on the NumPy site to THE ORDER OF THE EQUATIONS GIVEN
see what options are available IN THE EQUATION PANE
(http://www.numpy.org). Below the Initial values, you can
Below these two panes is where enter the start time, the end time
you define the rest of the options and the time step to use in the
for your problem. In the Initial solution. The critical time steps
values box, you can enter the initial box is usually left empty, so let’s

Figure 2. Once you finish defining the problem and run the integration, a result window
pops up with a graph of the integration.

26 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 26 2/19/15 9:21 AM


[ UPFRONT ]

Figure 3. You always can get a typeset display of your equations to verify what they
should look like.

leave it alone here. The first determination. The full output


step box is the size of the first check box will print out more useful
step. Usually, you should leave information about the integration in
this as 0 to allow for automatic the results spreadsheet.
determination. The minimum Once everything is entered, all
and maximum step size boxes set you need to do is click the Start
these variables that are used in icon, and the integration will be
the variable step size algorithm. calculated. If this is a system that
Typically, you should leave these you will want to work with over
as 0 as well to allow for automatic time, you can click on the menu

WWW.LINUXJOURNAL.COM / MARCH 2015 / 27

LJ251-March2015.indd 27 2/19/15 9:21 AM


[ UPFRONT ]

Figure 4. You can pull up all of the results of your integration and do further analysis.

item FileASave to save the model this graph window, so you can
to a file. This file format is an manipulate it just like any other
XML file, so you could edit it with matplotlib window. This includes
a text editor if you want. When panning, zooming or changing the
you are ready to do more work plot window. You also can save the
with it, you can load it by clicking resulting plot as an image file in
on FileAOpen. one of several different formats.
Once the calculations are done, Going back to the main window,
which may be fast for simple let’s look at some other available
problems, a results window will pop tools. Clicking on the Show
up (Figure 2). matplotlib handles EQUATIONS ICON POPS UP A WINDOW

28 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 28 2/19/15 9:21 AM


[ UPFRONT ]

Figure 5. You can generate a power spectrum of any column of your results.

WHERE YOU CAN SEE THE EQUATIONS column headers. Then, click on
typeset (Figure 3). Beside this icon the plot button to plot them in a
is the Results icon. Clicking on new window. You can get a power
that pops up a spreadsheet of all spectrum for any one column by
of the results from your integration selecting one of interest and clicking
&IGURE   4HE COLUMNS OF DATA on the Spectrum icon. This pops up
include the time, the value of two new windows, the first a power
y[0] and the step sizes, among spectrum of the column (Figure 5)
other things. You can select a and the second a spectrogram of the
couple columns by holding down column (Figure 6).
the Ctrl key and clicking on the The last tool available is a wavelet

WWW.LINUXJOURNAL.COM / MARCH 2015 / 29

LJ251-March2015.indd 29 2/19/15 9:21 AM


[ UPFRONT ]

Figure 6. You also can generate a spectrogram of your results.

transform. When you select a hopefully you will consider it


column, you can apply a continuous WHEN LOOKING AT /$% PROBLEMS
wavelet transform to the data. It provides a pretty simple interface
When you are done with Model to the tools available in Python to
Builder, you can save this data into SOLVE /$%S NUMERICALLY !LTHOUGH
A COMMA SEPARATED VALUES #36 FILE other more powerful tools are
from the spreadsheet window. Then, available, Model Builder fits into
you can import it into other tools, THE NICHE OF EXPERIMENTING QUICKLY
like R, to do even further analysis. WITH DIFFERENT EQUATIONS AND PLAYING
Now that you have seen the with ideas.
options available in Model Builder, —JOEY BERNARD

30 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 30 2/19/15 9:21 AM


'15HPCLinux-LinuxJnlMarchad.qxp_Layout 1 2/10/15 3:34 PM Page 1

C
fo F
12th Annual Re onf ree
rQ
2015 HPC FOR WALL STREET – ua gi ere
w
w lif str nc
w. G
fla o ied ati e
CLOUD TECHNOLOGY gg O
m nli nd
gm ne U
E on

April 6, 2015 (Monday) Roosevelt Hotel, NYC t.c - sers


Madison Ave and 45th St, next to Grand Central Station om .
/li
Plan to Attend: nu
x
2015 HPC for Wall Street will deliver
top-notch content and connections.
Low-cost conference at $295. save $100.
Full program, including lunch.
Free Conference Registration for quali- Global
Dave Weber Ken Barnes Bernard S Donefer Mike Blalock
Financial Services SVP Corp Dev, Options Associate Director, Global Sales Director,

fied end users. Register online as end


Segment Leader, Lenovo Information Technology Baruch College Intel

user.
Cloud Technology,Big Data, Low Latency, Networks, Data Cen-
ters, APIs, Scalability, cost savings for the global financial markets.
Leading Wall Street IT directors and vendor technology experts Paul Jameson Dave Malik Dino Vitale Harvey Stein
will speak on the program. Managing Director,
Global Fin Services,
Senior Director, Dir, Morgan Stanley Head of Credit Risk
Modeling,
Advanced Services, Quality Assurance &
Cisco Systems Cisco Systems Production Mgmt Bloomberg
Speakers will cover 2015 Cloud, HPC and the latest programs
to increase speed, put-through, and reduce costs.
Full conference program includes industry luncheon, general
sessions, drill down sessions, exhibits, post show receptions.
Don’t have time for the full Conference? Attend the free
Show. Register in advance at: www.flaggmgmt.com/linux Fadi Gebara Terry Keene Rob Krugman Lee Fisher
Sr Manager, CEO, VP Digital Strategy, VP Marketing, Redline
IBM Research iSys Broadridge Fin Sols Trading Solutions
2015 Sponsors

Jeremy Eder Matt Smith David B. Weiss Rick Aiere


Perf Engineering, Sol Architect, Sr Analyst, Architect Specialty,
Red Hat Red Hat Aite AIG

Shagun Bali Jeffrey Scheel Ed Turkel Charles Milo


Analyst, Senior Technical Staff, Mgr WW HPC Mkting, Enterprise Technical
TABB Group IBM Linux Tech Center Hewlett-Packard Specialist, Intel

Show & Conference:


Flagg Management Inc
353 Lexington Avenue,
Show Hours: Mon, April 6 8:00 - 4:00 New York 10016
Conference Hours: Mon, April 6 8:30 - 4:50 (212) 286 0333
fax: (212) 286 0086
flaggmgmt@msn.com
Visit: www.flaggmgmt.com/linux Davor Frank
Sr Solutions Architect,
Solarflare
Phil Albinus
Editor, Traders Maga-
zine, SourceMedia

LJ251-March2015.indd 31 2/19/15 9:22 AM


[ EDITORS' CHOICE ]

Nmap—Not Just EDITORS’


CHOICE
for Evil! ★
If SSH is the Swiss Army knife of the You don’t even have to have root
system administration world, Nmap access for that, and it’s as simple
is a box of dynamite. It’s really as specifying the network block you
easy to misuse dynamite and blow want to scan. For example, typing:
your foot off, but it’s also a very
powerful tool that can do jobs that nmap 192.168.1.0/24
are impossible without it.
When most people think of WILL SCAN THE ENTIRE RANGE OF 
Nmap, they think of scanning possible IP addresses on my local
servers, looking for open ports network and let me know which
to attack. Through the years, are pingable, along with which
however, that same ability is ports are open. If you’ve just
incredibly useful when you’re in plugged in a new piece of
charge of the server or computer hardware, but don’t know what
IN QUESTION 7HETHER YOURE TRYING IP address it grabbed via DHCP,
to figure out what kind of server Nmap is priceless. For example,
is using a specific IP address in the above command revealed this
your network or trying to lock on my network:
down a new NAS device, scanning
networks is incredibly useful. Nmap scan report for

Figure 1 shows a network scan ´TIVO-8480001903CCDDB.brainofshawn.com (192.168.1.220)

OF MY 1.!0 .!3 4HE ONLY THING ) Host is up (0.0083s latency).

use the unit for is NFS and SMB file Not shown: 995 filtered ports

sharing, but as you can tell, it has PORT STATE SERVICE

a ton of ports wide open. W ithout 80/tcp open http

Nmap, it would be difficult to figure 443/tcp open https

out what the machine was running. 2190/tcp open tivoconnect

Another incredibly useful way 2191/tcp open tvbus

to use Nmap is to scan a network. 9080/tcp closed glrpc

32 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 32 2/19/15 9:22 AM


Figure 1. Network Scan

This not only tells me the address .MAP GETS THIS MONTHS %DITORS
of my new T ivo unit, but it also Choice award. It’s not a new
shows me what ports it has open. program, but if you’re a Linux
Thanks to its reliability, usability user, you should be using it!
and borderline black hat abilities, —SHAWN POWERS

WWW.LINUXJOURNAL.COM / MARCH 2015 / 33

LJ251-March2015.indd 33 2/19/15 9:22 AM


COLUMNS
WORK THE SHELL

Let’s Play DAVE TAYLOR

Cards with
Acey-Deucey,
Part II
Dave adds the necessary code to turn a demo into a playable
game, complete with some rule variants.

In my last article, I started For the example above, there are


developing a simple card game four 6s, four 7s, 8s, 9s and 10s,
called Acey-Deucey, in which you MEANING THAT THERE ARE 
  
deal two cards face up, then bet   OR A  CHANCE THAT
on whether the next card is going the next card flipped up will indeed
to be between those two in rank be between the two exposed cards.
value. In other words, if a 5 of Make that 5 of diamonds an ace of
diamonds and a jack of spades diamonds, and the odds get crazy
were flipped up, the bet would be good: 80%. I’d take those odds!
whether the next card was going The math will factor into the
to be between a 6 and a 10. script because you actually can
I also dug into the math too, if have the game suggest what to
you missed it, because this is a great do based on the odds. The greater
game for understanding odds and the spread, the better the odds—
probability. Remember, any given card easy enough.
has a 1 in 52 chance of appearing, I ended my last article with the
and because two cards already have game being able to shuffle and deal
been exposed, that means any given three cards: two exposed and one
card actually has 1:50 odds. hidden. Running the program with

34 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 34 2/19/15 9:22 AM


COLUMNS
WORK THE SHELL

More important, it also means that the game


can identify situations where there’s no point
in betting, like when a 7 of diamonds and 8 of
clubs are dealt out.

just that code results in this: helpful, particularly knowing that the
dealCards function ensures that the
$ sh acey-deucey.sh two cards displayed are in order of
I've dealt: increasing rank, which means that this
Ace of Hearts is a darn helpful addition:
Queen of Diamonds
$ splitValue=$(( $rank2 - $rank1 ))

There’s not much to do yet, More important, it also means


because there’s no game logic, so that the game can identify situations
let’s add some. where there’s no point in betting,
like when a 7 of diamonds and 8
Turning the Code into a of clubs are dealt out. There are no
Playable Game cards that can be between them.
To start, let’s initialize and deal out This is added with a simple test:
the cards. With the highly mnemonic
function names already assigned, it’s if [ $splitValue -le 1 ] ; then

QUITE READABLE echo "No point in betting when you can't win!"

continue

initializeDeck fi

shuffleDeck

dealCards The third card already has


echo "Do you think the next card will be between? (y/n/q) " been “dealt” within the function
read answer dealCards , its rank calculated (as
$rank3 ) and its display name set (as
This is good for a start, but as I $cardname3 ). So, the test to see if the
mentioned earlier with the math new card is or isn’t between the two
discussion, it can be a bit more existing ranks is the next section of the

WWW.LINUXJOURNAL.COM / MARCH 2015 / 35

LJ251-March2015.indd 35 2/19/15 9:22 AM


COLUMNS
WORK THE SHELL

So you can pick three cards randomly out of the


deck, you can calculate their ranks and display
names, and you can prompt the user to guess
whether the next card will or won’t be between
the two, then test to see if they were right.

CODE REQUIRED AND IT TOO IS EASY won=$(( $won + 1 ))

else

if [ $rank3 -gt $rank1 -a $rank3 -lt $rank2 ] ; then # winner! echo "Bad betting strategy. You lose."

winner=1 fi

else

winner=0 You’ll notice that in this


fi implementation of Acey-Deucey, I’m
allowing the player to win if he or she
So you can pick three cards randomly bet the card won’t be between the
out of the deck, you can calculate their two, and it turns out that it isn’t. This
ranks and display names, and you can is probably too generous, because all
prompt the user to guess whether the you need to do is pick the more likely
next card will or won’t be between the scenario, which is to say any situation
two, then test to see if they were right. where the spread is six cards or less (like
What’s left? Scoring. And, that’s at the very beginning of this article).
done with the $won variable, which is 3TILL ITS NOT 6EGAS OR !TLANTIC #ITY ITS
incremented in a conditional statement just a shell script, right? So I’ll be nice.
that appears immediately after the test to If you’d rather not offer that option,
see if the third card is a $winner or not: simply change the message in the first
elif conditional code block and skip
if [ $winner -eq 1 -a "$answer" = "y" ] ; then incrementing the $won variable.
echo "You bet that it would be between the two and it is. All that’s left to do is to wrap the
You WIN!" entire code block in a big loop that’ll run
won=$(( $won + 1 )) FOREVER AND USE THAT STANDARD TECHNIQUE
elif [ $winner -eq 0 -a "$answer" = "n" ] ; then of shell script programmers worldwide:
echo "You bet that it would not be between the two and

it isn't. You WIN!" while [ /bin/true ] ; do

36 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 36 2/19/15 9:22 AM


COLUMNS
WORK THE SHELL

You probably wondered why /bin/true The spread is 3. Do you think the next card will

existed in Linux, didn’t you? So that’s be between them? (y/n/q) n

the first line of the main code block, I picked: 9 of Hearts

and let’s increment the $games You bet that it would not be between the two

variable in the last line of the block: and it isn't. You WIN!

I've dealt:

games=$(( games + 1 )) Ace of Hearts

7 of Spades

But there’s one more fragment The spread is 6. Do you think the next card will

needed, and that’s the test to see if be between them? (y/n/q) y

the user guessed that the third card I picked: 3 of Spades

would or would not be between the You bet that it would be between the two

two displayed cards, or if the user and it is. You WIN!

QUIT THE GAME )N THE LATTER SITUATION I've dealt:

it’s time to display some stats. That’s 7 of Spades

easy enough, and it turns out that 10 of Spades

you can just leave $answer alone The spread is 3. Do you think the next card will

for the yes/no answer: be between them? (y/n/q) q

You played 2 games and won 2 times.

if [ "$answer" = "q" ] ; then $

echo "You played $games games and won $won times."

exit 0 ! PERFECT SCORE .ICE ,AS 6EGAS


fi here I come! Q

)N FACT YOULL NEVER QUIT THE GAME Dave Taylor has been hacking shell scripts for more than 30
by falling out of the while loop, but years—really. He’s the author of the popular Wicked Cool
that makes sense since the conditional Shell Scripts (and just completed a 10th anniversary revision
test of /bin/true is, well, always true. to the book, coming very soon from O’Reilly and NoStarch
Stitch all these fragments together Press). He can be found on Twitter as @DaveTaylor and more
and you have a game, by George! generally at his tech site http://www.AskDaveTaylor.com.

$ sh acey-deucey.sh

I've dealt: Send comments or feedback via


6 of Hearts http://www.linuxjournal.com/contact
9 of Clubs or to ljeditor@linuxjournal.com.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 37

LJ251-March2015.indd 37 2/19/15 9:22 AM


COLUMNS
HACK AND /

Libreboot on KYLE RANKIN

an X60, Part I:
the Setup
Find out what Libreboot is and why you should dust off that old
ThinkPad and give it a fresh BIOS.

Recently I wrote a review and libreboot are great free software


for the Linux Journal Web site BIOS implementations, to get it on
on the Purism Librem 15 laptop MANY LAPTOPS REQUIRES HARDWARE
(http://www.linuxjournal.com/ BIOS chip flashing with pomona
content/purism-librem-15-review). clips—the kind of thing I wasn’t
The goal of this laptop is to provide a ready to brick a laptop to try. Like
piece of modern hardware that can run other privacy advocates, I turned
100% free software not just for the OS, to the old ThinkPad X60 laptop
but also all device drivers and firmware series. While it’s old, underpowered
up to and including the BIOS. At the and has a low-res screen by today’s
time I’m writing this, the last major standards, the keyboard is great and
sticking point along those lines for the more important, you could flash its
PROJECT IS THE )NTEL -ANAGEMENT %NGINE BIOS with coreboot or libreboot from
a proprietary piece of firmware that is within Linux itself—no hardware
REQUIRED TO BOOT UP MODERN SYSTEMS )N HACKING REQUIRED 3O THATS WHAT ) DID
that review, I wrote the following:
Although the Purism 15 laptop
It turns out it’s rather difficult to seems to be a viable choice for
have a fully free software laptop. those who want a free software
%VEN IF YOU CAN PICK HARDWARE THAT laptop, at the time of this writing,
can use free software drivers, there’s the crowdfunding campaign is still in
still that pesky BIOS. While coreboot process, and even after it completes,

38 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 38 2/19/15 9:22 AM


COLUMNS
HACK AND /

I’ve been able to find used ThinkPad X60 laptops


on auction sites as cheap as $30, so if you are
willing to live with some of the limitaions of
hardware that old, it is an inexpensive route to
a decent machine that runs only free software.

it will take some time until they ship. going to walk through the journey
Plus, a new laptop like that doesn’t that brought me to the X60 running
come cheap, and many people who Libreboot that I’m using to type this
may want a laptop that runs 100% column. In this first part, I discuss the
FREE SOFTWARE MAY NOT HAVE   setup, including what Libreboot is,
to spend on it. I’ve been able to what hardware it currently supports
find used ThinkPad X60 laptops on and some of the risks around flashing
auction sites as cheap as $30, so if your BIOS. If I haven’t scared you
you are willing to live with some of off by the end of this article, in
the limitations of hardware that old, future articles, I’ll cover how to
it is an inexpensive route to a decent download Libreboot and verify its
machine that runs only free software. integrity, how to flash the BIOS itself
The first time I attempted to flash in detail with sample script output
an X60 with coreboot, it was one of and how to modify the default GRUB
the more difficult things I’d done with bootloader. If you can’t wait until
Linux to the point that I wasn’t ever next month, a lot of my process
planning on writing it up in Linux is based on the excellent guide
Journal. More recently, I tried again, provided at https://github.com/
only this time with Libreboot—a bibanon/Coreboot-ThinkPads/wiki/
coreboot BIOS distribution that ThinkPad-X60.
has all of the proprietary software
removed. The process was greatly Free as in BIOS
simplified and automated to the point To understand Libreboot, it helps to
where I feel relatively comfortable understand coreboot first. Coreboot
recommending others try it (with a is an open-source BIOS replacement.
few caveats I’ll explain later). With coreboot, you can replace a
In my next couple articles, I’m proprietary BIOS with open-source

WWW.LINUXJOURNAL.COM / MARCH 2015 / 39

LJ251-March2015.indd 39 2/19/15 9:22 AM


COLUMNS
HACK AND /

Libreboot is a custom distribution of coreboot that


removes all proprietary software from the BIOS.

software on supported hardware image for my laptop and went through


with a minimal amount of proprietary a two-phase flash. In the end, I got it
firmware included to support things working; however, I needed to strip
like video hardware in the BIOS or the out and include the proprietary video
)NTEL -ANAGEMENT %NGINE ON NEWER firmware from my proprietary BIOS
hardware. Coreboot doesn’t currently to get any video at boot time—useful
support all hardware out there, when you want to select between hard
although the list continues to grow, drive and USB boot.
and you might be surprised to know Libreboot is a custom distribution of
that Chromebooks ship with coreboot coreboot that removes all proprietary
by default. To install coreboot on software from the BIOS. Instead of
much of the supported hardware, you proprietary BIOS boot selector, for
must use external hardware including instance, Libreboot boots straight into
a connector like an 8-pin Pomona its own GRUB menu that you can use
clip to reflash the BIOS chip. That’s to load your own underlying OS. In
pretty intense for a lot of people, but addition, Libreboot has automated a
fortunately, some hardware including lot of the difficult processes around
the X60, X60s, X60 tablet and T60 installing coreboot and provides
can be flashed completely in software. custom scripts and pre-build ROMs for
When I first attempted to flash an its officially supported hardware.
X60 with coreboot a few months ago, But, why would you want a free
the process involved disassembling the software BIOS? For those who
laptop to inspect the underside of the fully support the Free Software
motherboard with a magnifying glass Foundation and the principles of
so I could determine which of two BIOS free software, you don’t need any
chip types I had. I used that information further justification. Although I have
to hand-patch the flashrom software traditionally taken a more pragmatic
with custom code and compiled a approach to the free vs. open-source
special version just to unlock my BIOS. software debate, I’ve recently been
Then I downloaded, configured and more motivated to seek out free
compiled a custom coreboot BIOS software whenever I can find it as I

40 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 40 2/19/15 9:22 AM


COLUMNS
HACK AND /

explain in my Librem 15 review: limited. Among other reasons, this


is due to the fact that modern Intel
In the past, I didn’t care all that HARDWARE REQUIRES THE PROPRIETARY )NTEL
much if I had to use a binary blob -ANAGEMENT %NGINE FIRMWARE EVEN TO
to get a wireless card or video card boot. Although you may be able to get
working as long as it worked, and I Libreboot to work on other hardware,
definitely never cared that my BIOS at this point, only a few laptops are
was proprietary software. listed on its hardware compatibility
list (http://libreboot.org/docs/hcl/
Then the Snowden leaks happened. index.html#supported_list) as
The sheer depth and breadth of officially supported:
the loss of privacy motivated me to
step up my game in terms of overall Q Lenovo ThinkPad X60/X60s
security and focus on privacy. In the
past it would seem rather paranoid Q Lenovo ThinkPad X60 Tablet
to think that there might be some
sort of NSA-sanctioned spyware in a Q Lenovo ThinkPad T60
binary blob, firmware, or the BIOS.
After the Snowden leaks and the Q Apple MacBook1,1
SUBSEQUENT DISCLOSURES ABOUT THE
ANT catalog, these things stopped Q Apple MacBook2,1
seeming so far-fetched. I found
myself leaning more toward the You may find one major thing in
Stallman camp. One of the only common with all the laptops on this
ways to be truly sure that you don’t list: they are old. In most cases, we
have a backdoor on your system is are talking about 32-bit Intel Core
to be able to see the source code $UO PROCESSORS OR  BIT #ORE  $UOS
for all of it from the browser plugins in some cases (and the T60’s CPU
to the kernel drivers all the way to CAN BE REPLACED WITH A  BIT #05
the BIOS. apparently). That said, the X60 is a
decent piece of hardware with a solid
Supported Hardware keyboard and decent battery life,
Due to the fact that Libreboot avoids even if the CPU is slow and the screen
any proprietary firmware in the BIOS, resolution is low by today’s standards.
its hardware support is somewhat %VEN ON THIS LIST OF SUPPORTED

WWW.LINUXJOURNAL.COM / MARCH 2015 / 41

LJ251-March2015.indd 41 2/19/15 9:22 AM


COLUMNS
HACK AND /

hardware there are some exceptions. the initial bootstrapping flash phase. If
Although all X60s are supported, that happens but you were using one
only T60s that use Intel GPUs are of the Libreboot-supplied ROMs, all
supported, and those with ATI GPUs you should have to do is shut off the
are not. The Libreboot hardware machine, unplug the CMOS battery for
compatibility page has more a few seconds, reconnect it and power
information to help you figure out on your machine to get back to the
what’s supported and what isn’t. The original BIOS.
page also lists recommended Wi-Fi If you flash during the initial
chipsets that are known to work well bootstrapping phase with a custom
with Libreboot and Linux in general, ROM like I tried one time, lose power
AS THEY DONT REQUIRE ANY PROPRIETARY during the process, attempt this on
binary blobs to function. incompatible hardware or otherwise
encounter a worst-case scenario,
Risky Business you could end up with a completely
If it doesn’t already go without saying, unbootable machine. Because you
reflashing the BIOS on your laptop with can’t boot back to your OS, you
custom software is risky! Although I’ve can’t attempt to reflash, so you are
had success so far flashing a couple stuck with a bricked laptop unless
different X60s, I did temporarily brick you buy hardware that can flash your
one laptop when I got fancy and BIOS chip, such as a BusPirate or a
tried an initial flash with one of my Raspberry Pi running custom software.
own custom ROMs instead of one That said, if you have that hardware,
provided by Libreboot. For the most wire it properly and you remembered
part, the process is straightforward to back up your original BIOS first,
and automated, but as you’ll see in you should be able to restore your
my follow-up article that describes laptop to normal.
each step, many of the automated Although so far I’ve been successful
scripts call other software that output when I’ve stuck strictly to the
some pretty scary warnings and directions, there is still a possibility
errors during the process that you are you will brick your laptop, so if
supposed to ignore. you are particularly attached to
There are two primary ways you can your laptop and can’t risk it being
brick your laptop during the process. OUT OF SERVICE WHILE YOU ACQUIRE
First, you could have a bad flash during hardware flashing tools, you may

42 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 42 2/19/15 9:22 AM


COLUMNS
HACK AND /

want to reconsider going down this AND REQUIRES A NUMBER OF UNUSUAL


road. Again, you can get used X60s steps, most of the hard work already
relatively cheap on-line if you shop has been done for you, and in the end
around, so if you are concerned, you’ll have a trusted machine without
you may want to try this first with any proprietary firmware. Q
a sacrificial machine.
Kyle Rankin is a Sr. Systems Administrator in the San Francisco
Conclusion Bay Area and the author of a number of books, including The
Well, if I haven’t scared you off yet, I Official Ubuntu Server Book, Knoppix Hacks and Ubuntu Hacks.
hope you check out my next column He is currently the president of the North Bay Linux Users’ Group.
in this series where I jump right into
step-by-step instructions on how to
flash an X60 with Libreboot. Although Send comments or feedback via
THE PROCESS ISNT QUITE AS SIMPLE AS http://www.linuxjournal.com/contact
updating a traditional proprietary BIOS or to ljeditor@linuxjournal.com.

LINUX JOURNAL
on your
Android device
Download the app now on
the Google Play Store

www.linuxjournal.com/android
For more information about advertising opportunities within Linux Journal iPhone, iPad and
Android apps, contact John Grogan at +1-713-344-1956 x2 or ads@linuxjournal.com.

LJ251-March2015.indd 43 2/19/15 9:22 AM


COLUMNS
THE OPEN-SOURCE CLASSROOM

The Teeny SHAWN POWERS

Tiny $20 Tablet


What’s better than a pocket-sized Android tablet? One for $20.

For reasons other than “which do before I ever could order one. It seems
you like better”, my cell phone is an my demographic is tiny enough that
Apple iPhone. Mainly it’s because the it can’t support a line of devices.
rest of my family members use Apple Thankfully, my demographic is also
products, and I want to be able to fit pretty nerdy, so with a little research
into their environment. With three and hard work, I got a better solution
teenage daughters, it’s nice to run altogether—for $20.
“Find my iPhone” and see why they’re
running late. That leaves me with two My Prepaid Non-Phone
problems. First, there’s the ridicule The short version is that I bought a
and teasing from my geeky friends. prepaid Android phone and never
(You know who you are!) The second activated it. That version of the story
problem is that I really love Android leaves out some really important
apps for much of what I do on a and really cool details, however.
day-to-day basis. My Nexus 7 is too My end result is a pocket-sized
unwieldy to carry around all the time, Android device that I can use for
so I really need a tiny little Android listening to audiobooks via Bluetooth
tablet I can keep in my pocket. If the headset, make and receive calls,
roles were flipped, I could just buy play games, and sorta use for a GPS
an iPod Touch and be done with it. device while driving. The best part is
It turns out things aren’t so simple that my uber-micro-tablet really did
in the Android world. cost me only $20.
I was able to find the Samsung If you’re lazy, you can just buy
Galaxy Player in several sizes, but a prepaid phone off the shelf and
not only did they cost hundreds of never activate it. Most (but not
dollars, they also were discontinued all) will allow you to cancel the

44 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 44 2/19/15 9:22 AM


COLUMNS
THE OPEN-SOURCE CLASSROOM

If you do some research and don’t mind a little


hard work, however, you can get a cheap Android
device that does everything you want without any
nag screens or limitations.

activation screen and use the device want a powerhouse, but I wanted
without cell service. If you do some to be able to do things with the
research and don’t mind a little hard device. I wanted at least 2GB
work, however, you can get a cheap of RAM as well, but I ended up
Android device that does everything settling for 1GB.
you want without any nag screens
or limitations. I describe my process Q MicroSD expansion slot: this
here, and if it sounds like something is vitally important, because
interesting, you can do the same. prepaid phones generally come
with absurdly small amounts of
My Requirements internal storage.
I wanted my new anti-iPod to be
every bit as useful as the Samsung Q Bluetooth: the main purpose of
Galaxy Player would have been. this device will be to listen to
Here’s what I expected: audiobooks. For that, it needs
to work with my knockoff-brand
Q ! SMALL BUT NICE QUALITY SCREEN version of the Logitech HB-730.
I didn’t want a cheap plastic
screen that would haze over Q Must be rootable: this is as
with tiny scratches. Preferably, important as the MicroSD slot.
I wanted Gorilla Glass. 7ITH THE ADVENT OF !NDROID 
you need to have a rooted device
Q Wi-Fi: this seems obvious, but in order for applications to be able
with cheap prepaid phones, you to write to the SD card. I personally
never can tell. It’s always safest think it’s about the dumbest
to make sure! “feature” a new version of Android
could offer, but at least with root
Q At least a dual-core CPU: I didn’t access, it can be fixed.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 45

LJ251-March2015.indd 45 2/19/15 9:22 AM


COLUMNS
THE OPEN-SOURCE CLASSROOM

Q Must be affordable. I already have great battery life along with really
a phone (the iPhone), so I have to great cameras. You currently can
be able to convince my wife that pick up this device for around $60,
it’s not wasteful to buy a prepaid and for the hardware you get,
phone I never plan to activate. I’m guessing Boost is losing some
(APPY WIFE  HAPPY LIFE money on every sale.

My New Non-Phone 3. LG Realm from Boost Mobile: the


I spent a very, very long time Realm is what I ended up buying
researching what phone to purchase. (Figure 1). The specs are a step
Since what I was proposing goes DOWN FROM THE 6OLT BUT ) WAS ABLE
against everything the prepaid to get the device for $19.99 from
vendors stand for, it’s not like I could "EST "UY AT THE END OF  IN THE
check their Web sites to see if the “last chance to get Black Friday
phones were rootable or if they’d Sale Prices” sale. That sale probably
work without activation. I considered still is running; it seems that’s how
several models: Black Friday sales work nowadays.

 -OTOROLA -OTO ' FROM 6ERIZON Phone models change all the time.
and Boost Mobile: the Moto G is Rather than make decisions based
a pretty decent-looking phone, on my findings from a few months
and it has a beautiful screen. ago, I urge you to look for the latest
Unfortunately, although it has and greatest (or cheapest!) prepaid
8GB of onboard storage, it lacks options out there, and make sure
a MicroSD expansion slot. It’s also THEY MEET YOUR LIST OF REQUIREMENTS
around $80, which is reasonable I can’t stress enough how important
considering how nice of a device it is for the phone to be rootable
it is, but without that SD slot, it’s though, so do at least that much
more than I was willing to pay. research before buying one.

 ,' 6OLT FROM "OOST -OBILE THIS The Rooting


phone is probably what I’d buy Sometimes the hardest part of the
if I were going to buy another PROCESS IS TO GET OUT OF THE h!#4)6!4%
device right now. It checks all -% ./7v SCREEN 7ITH ENOUGH
the boxes above, and it has really button pressing, I was able to put the

46 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 46 2/19/15 9:22 AM


COLUMNS
THE OPEN-SOURCE CLASSROOM

Figure 1. Oddly,
a replacement
battery for this
phone costs more
than the phone
itself. At $19.99,
you can’t go
wrong!

activation screen in the background. http://towelroot.com from the


%VERY TIME THE PHONE BOOTED HOWEVER phone’s browser and installing the
it had the same annoying screen trying tr.apk file. As long as your phone is
to force me to activate. Therefore, the supported, it’s literally 2–3 clicks, and
very first thing I recommend doing is your phone is rooted. Then install
rooting the phone. SuperSU from the Google Play store,
Usually, that’s as simple as visiting and your phone is ready to hack.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 47

LJ251-March2015.indd 47 2/19/15 9:22 AM


COLUMNS
THE OPEN-SOURCE CLASSROOM

It’s important to note that rooting I had to rename /system/app/


a phone is not the same as installing LGDMSClient.apk to /system/app/
a third-party ROM. Although it’s dead LGDMSClient.apk.bak.
simple to root most phones, installing
something like Cyanogenmod is far  2EBOOT THE PHONE AND SEE IF IT
more difficult, and often it’s not worked. If it did, celebrate. If not,
possible even if the phone is rooted. do some more googling, or just
Thankfully, once the phone is rooted, educated guessing, and try again.
the existing ROM can be made to There is some danger here, but as
function a little nicer. Getting rid of long as you’re not deleting files,
the nag screens is the first obstacle just renaming them, most bad
in that journey. guesses can be reversed.

Stop the Nags! Other Anti-iPod Tweaks


Once your phone is rooted, it’s time Once your phone is working, and
to start looking for the applications you’re able to reboot it without the
that are doing all the nagging. frustrating nag screens, head over to
Unfortunately, this will take some the settings app. Here is where you
googling, some guessing and a can disable all cellular data radios.
little bit of luck. The process is itself Since you’re not going to activate the
pretty straightforward: phone with cell service, it will save
some serious battery power if you
1. Download a root-enabled file disable the radios entirely. Depending
manager app like Root Browser on your model, you may have to
or something similar. DISABLE ',4% AND ' SEPARATELY
The one frustration I have is that
2. Figure out what app(s) are try as I might, I’ve not been able to
responsible for the activation nag remove the cellular radio icon from
screens. Basically, google your the top of the phone (Figure 2).
phone’s model along with “disable There are some apps in the Google
activation screen” or something Play store to remove icons, but they
like that. remove the W i-Fi icon too, and that
doesn’t help me at all. Oh well,
3. Rename the apk files to add it’s a small price to pay. Plus, it’s a
.bak at the end. In my case, great way for me to keep track of

48 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 48 2/19/15 9:22 AM


COLUMNS
THE OPEN-SOURCE CLASSROOM

Figure 2. Although
the cellular radio
icon bothers
me, it’s the only
frustration I
haven’t been able
to eliminate!

Sprint coverage in my area. If it which would make my new mini-


ever gets strong enough, I’ll tablet even more useful on the road.
probably invest in a Karma router One last recommendation I have
(http://www.yourkarma.com), is to download one of the “SD Fix”

WWW.LINUXJOURNAL.COM / MARCH 2015 / 49

LJ251-March2015.indd 49 2/19/15 9:22 AM


COLUMNS
THE OPEN-SOURCE CLASSROOM

apps from the Google Play store. a year ago, I wouldn’t count on it
7ITH THE ADVENT OF !NDROID  THE working forever. Any SIP provider will
SD card isn’t writable by apps like work with CSipSimple, however, so
FolderSync, and as such, it makes EVEN IF THE FREE 'OOGLE 6OICE OPTION
managing audiobooks or MP3 files through Simonics stops working, you
really difficult. With a rooted phone, can get the prepaid phone working
it’s another two-click solution to make without paying the cellular provider.
your SD card functional again. I still
can’t believe Google crippled Android GPS!
 LIKE THAT 4HANKFULLY ROOT ACCESS It’s hard to buy an Android device
and Linux can save the day. that doesn’t come with GPS built in.
Although the lack of cellular radio
Here’s My Number, Call Me Maybe means you can’t do real-time map
If the cell radio icon bothers me, just downloading on the road (unless you
imagine how much it bothers me to have mobile Wi-Fi or a hotspot on
have a phone with microphone and your actual cell phone), it’s simple to
speaker, but no phone service. I know, use your new mini-tablet as a GPS
I said I wasn’t looking for a phone, device with a little bit of planning.
but I have OCD, so that unused Google Maps allows you to download
HARDWARE REALLY ANNOYS ME %NTER 3)0 map data for specific areas locally to
Some phones come with a firmware the device. This doesn’t work great
that supports Android SIP calling out for long trips, because grabbing all
of the box. Most prepaid phones, those “map sections” is tedious, but
however, disable that feature because for short trips to unknown locations
they want you to use their service. it works well.
It makes sense. Thankfully, you can There are also several off-line
download a third-party app called GPS apps available in the Google
CSipSimple and add complete Wi-Fi- Play store, and although most cost
based SIP calling to your phone. It money, they’re cheaper than buying
even integrates with the native a standalone GPS unit at the store.
dialer application, so you use it If I’m being completely honest, I still
like a regular phone! I’m still using use a Garmin GPS for long trips, but
'OOGLE 6OICE SERVICE THROUGH that’s probably because I’m old and
http://www.simonics.com, but because don’t always trust technology to
that ability was supposed to stop almost work as expected.

50 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 50 2/19/15 9:22 AM


LJ251-March2015.indd 51 2/19/15 9:22 AM
COLUMNS
THE OPEN-SOURCE CLASSROOM

What Else? Really, I’d do anything you can do


For me, having a tiny Android with any other Android device, but
device that lets me sync audiobooks without the guilt of spending a ton of
with FolderSync and play with money. Having a $20 Android device
the Listen app is all I could want capable of doing so many things really
and more. That doesn’t mean I makes it easy to come up with fun
couldn’t think of more things to projects. It’s also nice to be able to
do with a cheap Android device, mount one in the car and just leave it
however. I actually bought another there. With FolderSync, it will download
LG Realm when they were on sale, any new media when you’re parked in
and I have lots of plans for it— the garage, and you’ll never have to
things like: take your phone out of your pocket!
)M REALLY QUITE HAPPY 3AMSUNG
Q A Plex player for watching movies. discontinued its Galaxy Player devices.
I worry that if they were available, I
Q A really cheap IP camera (birdcam!) might have purchased one. I’m so much
using the IPWebcam app. happier with my $20 tablet than I would
have been with a $300 media player. If
Q Music player for the bathroom you have a need for another Android
counter, connected to speakers. device in your life, but don’t want to
spend a fortune, I urge you to check out
Q Tiny gaming device for bored the prepaid phone options out there.
children who visit. It’s surprising what $20 will buy! Q

Q Cheap Skype/Hangout device to Shawn Powers is the Associate Editor for Linux Journal.
give someone I want to keep in He’s also the Gadget Guy for LinuxJournal.com, and he has an
contact with. interesting collection of vintage Garfield coffee mugs. Don’t let
his silly hairdo fool you, he’s a pretty ordinary guy and can be
Q Surprisingly affordable alarm clock reached via e-mail at shawn@linuxjournal.com. Or, swing by
for my nightstand. the #linuxjournal IRC channel on Freenode.net.

Q Universal XBMC/Kodi remote for all


our televisions. Send comments or feedback via
http://www.linuxjournal.com/contact
Q Wi-Fi testing tool (WiFi Analyzer). or to ljeditor@linuxjournal.com.

52 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 52 2/19/15 9:22 AM


LJ251-March2015.indd 53 2/19/15 9:22 AM
NEW PRODUCTS

Gumstix Inc.’s Geppetto


Gumstix Inc. is so proud of the embedded
systems it designed with its home-grown
Geppetto design tool that it wants the
wider world to enjoy similar benefits.
Gumstix calls the new Geppetto 2.0 the
most advanced version of the company’s
on-line build-to-order tool for designing
custom-embedded Linux systems. This new
iteration of Geppetto introduces Tux-approved recommended mappings for buses,
ensuring optimal compatibility between customer-created hardware and standard
Linux images. In addition, version 2.0 offers an expanded module selection, improved
dimensioning, faster UI and video tutorials. As part of the Geppetto announcement,
'UMSTIX ALSO ANNOUNCED THE 'EPPETTO DESIGNED !ERO#ORE4-  -ICRO !ERIAL 6EHICLE
#ONTROL "OARD AND THE 'EPPETTO DESIGNED 0EPPER $6) $ SINGLE BOARD COMPUTER
http://www.gumstix.com

Investintech.com’s Able2Extract PDF


Converter
)TS NOT A STRETCH TO CALL )NVESTINTECHCOMS !BLE%XTRACT  0$& #ONVERTER
THE h3WISS ARMY KNIFEv OF 0$& CONVERTERS .OT ONLY IS !BLE%XTRACT ABLE
to convert PDFs to a wide range of formats accurately, but it also features
THE UNIQUE ABILITY TO WORK ACROSS ,INUX 5BUNTU AND &EDORA -AC /3 8
AND 7INDOWS PLATFORMS )NVESTINTECHCOM NOTES THE ABILITY OF !BLE%XTRACT
to maintain intact all aspects—images, colors, formatting and fonts—
regardless of file format. Supported formats include converting PDF to
/PEN/FFICEORG -3 /FFICE !UTO#!$ %XCEL AND COMMONLY USED IMAGE
formats. The upgrade version 9 adds secure PDF creation, improved custom
0$& TO %XCEL CONVERSION AND AN IMPROVED '5) AND OVERALL USER EXPERIENCE
http://www.investintech.com

54 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 54 2/19/15 9:22 AM


NEW PRODUCTS

Linutop XS
Until the era of the Linutop computer, the word
minuscule has not been a common descriptor for
a full-fledged PC. That word nevertheless hits the
NAIL SQUARELY ON THE HEAD TO DESCRIBE THE NEW
Linutop XS, a truly tiny Linux computer designed
to reduce TCO from shipping to deployment,
operation and maintenance. As Linutop’s smallest and most energy-efficient computer
to date, the Linutop XS weighs a mere 3.3 ounces (92 g), measures about the size
OF A TYPICAL PLAYING CARD AND OPERATES ON ONLY  6OLTS AND  7ATTS ,INUTOP SAYS
that the Linutop XS comes loaded with Debian Weezy and ready-to-use software,
including Libre Office and Linutop Kiosk, making it an ideal system for a wide range
of applications in business, government, education and the home.
http://www.linutop.com

JetBrains’ Upsource
The idea for JetBrains’ new team
collaboration tool for developers,
called Upsource, originally came
from the intention to make a
TOTALLY DIFFERENT TOOL )NTELLI* )$%!
available from both the desktop
and the Web. The final result is Upsource 1.0, a new Web-based team collaboration
tool that helps developers read, browse and review code maintained in Git, Mercurial,
Subversion and/or Perforce repositories. Both a repository browser and a code-review
tool, Upsource 1.0 provides instant read access to code developed throughout an
ORGANIZATION AND HELPS IMPROVE CODE QUALITY BY ENABLING EASY CODE REVIEW *ET"RAINS
ADDS THAT THANKS TO PLATFORM SHARING WITH THE )NTELLI* )$%! )$% FOR *AVA *AVA TEAMS
enjoy an additional advantage. Upsource boasts in-depth knowledge of Java code and
is able to execute server-side static code analysis on Java projects, as well as provide
code-aware navigation and smart search for code usages.
http://www.jetbrains.com/upsource

WWW.LINUXJOURNAL.COM / MARCH 2015 / 55

LJ251-March2015.indd 55 2/19/15 9:22 AM


NEW PRODUCTS

Corsair Flash Voyager Slider Series


X1 and X2
4HE NEW &LASH 6OYAGER 3LIDER 8 AND 8 FAMILIES OF 53"  &LASH
drives expand Corsair’s already formidable arsenal of memory
products. Combining the speed of USB 3.0 with the functionality of
A CAP LESS 53" DRIVE THE &LASH 6OYAGER 3LIDER 8 AND 3LIDER 8 SERIES
share a sleek, glossy design that allows the USB cap to slide back
conveniently into the drive housing, says Corsair. The company added that the Slider
8 IS AVAILABLE IN '" '" '" '" AND '" CAPACITIES AND THANKS TO ITS
USB 3.0 interface, is able to reach read speeds of up to 130MB/s. Meanwhile, Slider
X2 knocks the performance up a level with read speeds of 200MB/s in capacities of
'" '" '" AND '" "OTH #ORSAIR DRIVE FAMILIES ARE COMPATIBLE WITH ,INUX
Windows and Mac OS X, and they also are fully USB 2.0-backward compatible.
http://www.corsair.com

Regina O. Obe and Leo S. Hsu’s


PostGIS in Action, 2nd ed.
(Manning Publications Co.)
Hybrid GIS and Linux geeks know that the open-source
0OST')3 GIVES SUPPORT FOR GEOGRAPHIC OBJECTS TO 0OSTGRE31,
allowing the relational database to serve as the back end
for ArcGIS, GRASS GIS and other geospatial programs. The
new 2nd edition of PostGIS in Action from Regina O. Obe
AND ,EO 3 (SU TEACHES READERS OF ALL LEVELS TO WRITE SPATIAL QUERIES THAT SOLVE
real-world problems. Obe and Hsu start by getting readers’ feet wet with a
background in vector-, raster- and topology-based GIS, followed by a tutorial
IN ANALYZING VIEWING AND MAPPING DATA 2EADERS LEARN HOW TO OPTIMIZE QUERIES
for maximum speed, simplify geometries for greater efficiency, analyze rasters,
vectorize rasters, better manage data utilizing topologies and create custom
FUNCTIONS 4HE BOOK COVERS 0OST')3  AND  0OSTGRE31,   AND 
features and shows how to integrate PostGIS with other GIS tools.
http://manning.com

56 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 56 2/19/15 9:22 AM


NEW PRODUCTS

Mahesh Venkitachalam’s Python


Playground (No Starch Press)
Putting the subtitle Geeky Weekend Projects for the Curious Programmer onto
a book is a sure way to charm one’s way onto these geek-friendly Linux Journal
pages. The main title of said book is Python Playground A NEW BOOK FROM -AHESH 6ENKITACHALAM
and irreverent publisher No Starch Press. No Starch describes the book as “a collection of fun
programming projects that will inspire you to new heights as a Pythonista”. Readers will learn
to use Python for all kinds of playful purposes—for example, to manipulate images, build
simulations and interact with hardware using Arduino and Raspberry Pi. As readers work through
each project, they power up their programming skills and learn how to leverage external libraries
for specialized tasks, how to break problems into smaller, solvable pieces and how to translate
an algorithm into code. The fun projects include an autostereogram generator, an ASCII art
maker, a Conway’s Game of Life simulator, a ray casting volume renderer and an Arduino rig.
http://www.nostarch.com

Deciso OPNsense Firewall


$ECISO "6 IS A .ETHERLANDS BASED MANUFACTURER OF NETWORKING
EQUIPMENT THAT DEVELOPED AND RECENTLY RELEASED /0.SENSE A NEW
open-source firewall that reportedly “combines the best of open-
source and closed-source firewalls”. Deciso adds that OPNsense
brings the rich feature set of commercial offerings with the
benefits of open and verifiable sources, combined with a simple, two-clause BSD license. The latter
permits companies to create a branded firewall based on OPNsense, extend its features, or even
create a fork and build upon the same codebase. Key features of OPNsense include load balancing,
high availability and captive portal. The easy-to-use Bootstrap-based GUI makes configuring and
managing the firewall a comfortable task for administrators. The kicker, boasts Decisio, is that all
sources and build tools are freely available without special clauses and without licensing costs. The
company also puts a great deal of value on the community surrounding OPNsense, which it says will
give users, developers and businesses a friendly, stable and transparent environment.
http://www.opnsense.org and http://www.deciso.com

Please send information about releases of Linux-related products to newproducts@linuxjournal.com or


New Products c/o Linux Journal, PO Box 980985, Houston, TX 77098. Submissions are edited for length and content.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 57

LJ251-March2015.indd 57 2/19/15 9:22 AM


FEATURE Using Hiera with Puppet

USING
HIERA
WITH
PUPPET A GUIDE TO USING HIERA WITH PUPPET,
SEPARATING CODE FROM DATA
AND ENCRYPTING PASSWORDS
AND CERTIFICATES.

SCOTT LACKEY

58 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 58 2/19/15 9:22 AM


W
ith Hiera, you can credentials and other sensitive data,
externalize your systems’ which I talk about later in this article.
configuration data and Puppet node data originally was
easily understand how those values managed through node inheritance,
are assigned to your servers. With which is no longer supported, and
that data separated from your Puppet SUBSEQUENTLY THROUGH USING A PARAMSPP
code, you then can encrypt sensitive module subclass. Before Hiera, it was
values, such as passwords and keys. necessary to modify the params.pp
Separating code and data can be module class locally within the
tricky. In the case of configuration MODULE WHICH FREQUENTLY DAMAGED
management, there is significant value the re-usability of the module.
in being able to design a hierarchy of params.pp still is used in modules
data—especially one with the ability today, but as of Puppet version 3,
to cascade through classifications Hiera is not only the default, but also
of servers and assign one or several the first place checked for variable
options. This is the primary value that values. When a variable is defined
Hiera provides—the ability to separate in both Hiera and a module, Hiera
the code for “how to configure the takes precedence by default. As you’ll
/etc/ntp.conf” from the values that see, it’s easy to use a module with
define “what ntp servers should each params.pp and store some or all of
node use”. In the most concise sense, the variable data in Hiera, making it
Hiera lets you separate the “how” easy to migrate incrementally.
from the “what”. To get started using Hiera with your
The idea behind separating code existing Puppet 3 implementation, you
and data is more than just having a won’t have to make any significant
cleaner Puppet environment; it allows changes or code migrations. You need
engineers to create more re-usable only a hierarchy file for Hiera and a
Puppet modules. It also puts your yaml file with a key/value pair. Here is
variables in one place so that they too an example of a Hiera hierarchy:
can be re-used, without importing
manifests across modules. Hiera’s use hiera.yaml:
cases include managing packages
and versions or using it as a Node :backends:
Classifier. One of the most compelling - yaml
use cases for Hiera is for encrypting :yaml:

WWW.LINUXJOURNAL.COM / MARCH 2015 / 59

LJ251-March2015.indd 59 2/19/15 9:22 AM


FEATURE Using Hiera with Puppet

:datadir: /etc/puppet/hieradata recommend, which employs a


:hierarchy: fact assigned to all nodes called
- "node/%{::fqdn}" @env from within facter. This @env
- "environment/%{::env}/main" value can be set on the hosts either
- "environment/%{::env}/%{calling_module}" BASED ON &1$. OR TAGS IN %# OR
- defaults elsewhere, but the important thing
is that this is the separation of one
And a yaml file: large main.yaml file into directories
named prod, dev and so on, and,
/etc/puppet/hieradata/environment/prod/main.yaml: therefore, the initial separation of
--- Hiera values into categories.
$nginx::credentials::basic_auth: 'password' The second component of this
specific example is a special Hiera
Hiera can have multiple back ends, variable called %{calling_module} .
but for now, let’s start with yaml, 4HIS VARIABLE IS UNIQUE AND RESERVED
WHICH IS THE DEFAULT AND REQUIRES NO for Hiera to indicate that the yaml
additional software. The :datadir: filename to search will be the same as
is just the path to where the hierarchy the Puppet module that is performing
search path should begin, and is the Hiera lookup. Therefore, the
usually a place within your Puppet way this hierarchy will behave when
configuration. The :hierarchy: looking for a variable in Puppet is like:
section is where the core algorithm of
how Hiera does its key/value lookups $nginx::credentials::basic_auth
is defined. The :hierarchy: is
something that will grow and change First, Hiera knows that it’s looking
over time, and it may become much in /etc/puppet/hieradata/node for a file
more complex than this example. NAMED HOSTNAMEDOMAINTLDYAML
Within each of the paths defined and for a value for
in the :hierarchy: , you can nginx::credentials::basic_auth .
reference any Puppet variable, even If either the file or the variable
$operatingsystem and $ipaddress , isn’t there, the next step is to
if set. Using the %{variable} syntax look in /etc/puppet/hieradata/
will pull the value. ENVIRONMENTPROD\STAGE\DEV
This example is actually a special main.yaml, which is a great way
hierarchical design that I use and to have one yaml file with most

60 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 60 2/19/15 9:22 AM


of your Hiera values. If you have a Hiera will override it. This kind of
lot of values for the nginx example Hiera lookup is called Automatic
and you want to separate them for Parameter Lookup and is one of
manageability, you simply can move several ways to pull data from Hiera,
them to the /etc/puppet/hieradata/ but it’s by far the most common in
ENVIRONMENTPROD\STAGE\DEV practice. You also can specify a Hiera
nginx.yaml file. Finally, as a default, lookup with:
Hiera will check for the value in
defaults.yaml at the top of the modules/nginx/manifests/credentials.pp

hieradata directory.
Your Puppet manifest for this lookup
should look something like this: class nginx::credentials (

basic_auth = hiera('nginx::credentials::basic_auth'),

modules/nginx/manifests/credentials.pp ){}

These will both default to a priority


class nginx::credentials ( lookup method in the Hiera data files.
basic_auth = 'some_default', This means that Hiera will return the
){} value of the first match and stop
looking further. This is usually the
This class, when included, will only behavior you want, and it’s a
pull the value from Hiera and can reasonable default. There are two
be used whenever included in your lookup methods worth mentioning:
manifests. The value set here of hiera_array and hiera_hash .
some_default is just a placeholder; hiera_array will find all of the
Hiera will override anything set in a matching values in the files of the
parameterized class. In fact, if you hierarchy and combine them in an
have a class you are thinking about array. In the example hierarchy, this
converting to pull data from Hiera, would enable you to look up all values
just start by moving one variable for a single key for both the node
from the class definition in {} to a and the environment—for example,
parameterized section in (), and adding an additional DNS search
Puppet will perform a Hiera lookup path for one host’s /etc/resolv.conf.
on that variable. You even can leave To use a hiera_array lookup, you
the existing definition intact, because must define the lookup type explicitly

WWW.LINUXJOURNAL.COM / MARCH 2015 / 61

LJ251-March2015.indd 61 2/19/15 9:22 AM


FEATURE Using Hiera with Puppet

(instead of relying on Automatic * default


Parameter Lookup): :backends:
* psql
modules/nginx/manifests/credentials.pp :psql:
:connection:
:dbname: hiera
class nginx::credentials ( :host: localhost
basic_auth = hiera_array('nginx::credentials::basic_auth'), :user: root
){} :password: password

A hiera_hash lookup works in the You can do lookups on a local


same way, only it gathers all matching Postgres installation with a single
values into a single hash and returns database called hiera with a single
that hash. This is often useful for an table called config with three
advanced create_resources variable COLUMNS 0ATH +EY AND 6ALUE
import as well as many other uses in
an advanced Puppet environment. path key value

Perhaps Hiera’s most powerful


feature is the ability to pull data 'environment/prod' 'nginx::credentials::basic_auth' 'password'

from a variety of back-end storage


technologies. Hiera back ends are This is extremely useful if you want
too numerous to list, but they include to expose your Hiera data to custom
JSON, Redis, MongoDB and even HTTP in-house applications outside Puppet,
to create a URL-driven Puppet value or if you want to create a DevOps
API. Let’s take a look at two useful Web console or reports.
back ends: Postgres and hiera-eyaml. Storing credentials in Puppet
4O START WITH THE PSQL BACK END YOU modules is a bad idea. If you store
NEED TO INSTALL THE HIERA PSQL GEM ON credentials in Puppet and your
your Puppet master (or each node if manifests on an external code
you’re using masterless Puppet runs repository, you’re not only unable to
with Puppet apply), with a simple share those manifests with developers
hiera.yaml file of: with less-secure access, but you’re
obviously exposing vital security
:hierarchy: data outside the organization, and
* 'environment/%{env}' possibly in violation of various types

62 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 62 2/19/15 9:22 AM


of compliance. So how do you encrypt :hierarchy:

sensitive data in Puppet while keeping - "node/%{::fqdn}"

your manifests relevant and sharable? - "environment/%{::env}/main"

The answer is with hiera-eyaml. - "environment/%{::env}/%{calling_module}"

Tom Poulton created hiera-eyaml * defaults

to allow engineers to do just that:


encrypt only the sensitive string of To encrypt values, you need only the
data inside the actual file rather than public key, so distribute it to anyone
encrypting the entire file, which also who needs to create encrypted values:
can be done with hiera-gpg (a very
useful encryption gem but not covered $ eyaml encrypt -s 'password'
in this article).
To get started, install the hiera-eyaml This will generate an encrypted
gem, and generate a keypair on the block that you can add as the value in
Puppet master: any yaml file:

$ eyaml createkeys main.yaml:

nginx::credentials::user: slackey #cleartext example value

Then move the keys to a secure nginx::credentials::basic_auth : > #encrypted example value

location, like /etc/puppet/secure/keys. ENC[PKCS7,Y22exl+OvjDe+drmik2XEeD3VQtl1uZJXFFF2Nn

Your hiera.yaml configuration should /HjZFXwcXRtTlzewJLc+/gox2IfByQRhsI/AgogRfYQKocZg

look something like this: IZGeunzwhqfmEtGiqpvJJQ5wVRdzJVpTnANBA5qxeA==]

hiera.yaml: %DITING ENCRYPTED VALUES IN PLACE


--- is one of the coolest features of the
:backends: hiera-eyaml back end. eyaml edit
- eyaml opens a copy of the eyaml file in your
- yaml editor of choice and automatically
:yaml: decrypts all of the values in the file.
:datadir: /etc/puppet/hieradata Here you can modify the values just
:eyaml: as though they were plain text. When
:datadir: /etc/puppet/hieradata you exit the editor by saving the file,
:extension: 'yaml' # <- so all files can be named .yaml it automatically encrypts all of the
:pkcs7_private_key: /path/to/private_key.pkcs7.pem modified values and saves the new
:pkcs7_public_key: /path/to/public_key.pkcs7.pem file in place. You can see that the

WWW.LINUXJOURNAL.COM / MARCH 2015 / 63

LJ251-March2015.indd 63 2/19/15 9:22 AM


FEATURE Using Hiera with Puppet

unencrypted plain text is marked encrypted files in a separate repository,


to allow the eyaml tool to identify perhaps in a different Git repository.
each encrypted block, along with the Only the private keys need to be
encryption method that originally protected on the Puppet master. I
was used. This is used to make sure also recommend having separate
that the block is encrypted again only keys for each environment, as this
if the clear text value has changed can give more granular control over
and is encrypted using the original who can decrypt different datafiles in
encryption mechanism: Hiera, as well as even greater security
separation. One way to do this is to
nginx::credentials::user: user1 name the keys with the possible values
nginx::credentials::basic_auth : DEC(1)::PKCS7[very secret password]! for the @env fact, and include that in
the path of the hierarchy. You’ll need
Blocks and strings of encrypted text to encrypt values with the correct key,
can get rather onerous once you have and this naming convention makes it
more than a hundred entries or so. easy to tell which one is correct:
Because these yaml files are meant to
be modified by humans directly, you :pkcs7_private_key: /path/to/private_key.pkcs7.pem-%{::env}

want them to be easy to navigate. In :pkcs7_public_key: /path/to/public_key.pkcs7.pem-%{::env}

my experience, it makes sense to keep


your encrypted values in a separate When using Hiera values within
file, such as a secure.yaml, with a Puppet templates, either encrypted or
hierarchy path of: not, you must be careful to pull them
into the class that contains the templates
:hierarchy: instead of calling the values from
- "node/%{::fqdn}" within the template across classes—for
- "environment/%{::env}/secure" example, in the template mytest.erb in a
- "environment/%{::env}/main" module called mymodule:
- "environment/%{::env}/%{calling_module}"
mytest.erb:

This isn’t necessary, as each value ...

is encrypted individually and can be username: user1

distributed safely to other teams. It passwd: <%= scope.lookupvar('nginx::credentials::basic_auth') %>

may work well for your environment, ´#don't do this

however, because you can store the ...

64 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 64 2/19/15 9:22 AM


LJ251-March2015.indd 65 2/19/15 9:22 AM
FEATURE Using Hiera with Puppet

Puppet may not have loaded a value into mytest.erb:


nginx::credentials::basic_auth ...
yet because of the order of username: user1
operations. Also, if you are using the passwd: <%= @basic_auth %>
%calling_module Hiera variable, the
calling module in this case would be You’re now ready to start introducing
mymodule, and not nginx, so it would encrypted Hiera values gradually into
not find the value in the nginx.yaml your Puppet environment. Maybe after
file, as one might expect. you separate data from your Puppet
To avoid these and other issues, code, you can contribute some of your
it’s best to import the values into the modules to the PuppetForge for others
mymodule class and assign local values: to use! Q

mymodule.pp: Scott Lackey is a 17-year engineering veteran and Sr. DevOps


class mymodule { Engineer for Salesforce.com. He’s passionate about helping
include nginx::credentials companies migrate to the cloud and mentoring prospective
$basic_auth = "${nginx::credentials::basic_auth}" DevOps engineers. He lives in Los Angeles with his dachshund
file { '/etc/credentials/boto_cloudwatch.cfg': Zelda. Reach him at sudosudash@gmail.com.
content => template ("mymodule/mytest.erb"),

Send comments or feedback via


And then reference the local value http://www.linuxjournal.com/contact
from the template: or to ljeditor@linuxjournal.com.

Resources
Docs—Hiera 1 Overview: https://docs.puppetlabs.com/hiera/1

“First Look: Installing and Using Hiera”:


http://puppetlabs.com/blog/first-look-installing-and-using-hiera

TomPoulton/hiera-eyaml: https://github.com/TomPoulton/hiera-eyaml

dalen/hiera-psql: https://github.com/dalen/hiera-psql

“Encrypting sensitive data in Puppet”: http://www.theguardian.com/info/developer-blog/


2014/feb/14/encrypting-sensitive-data-in-puppet

66 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 66 2/19/15 9:22 AM


Interested in Site Reliability Engineering?

SREcon is back!
SREcon15
MARCH  16–17,  2015  
SREcon15EUROPE
MAY  14–15,  2015  
SANTA  CLARA,  CALIFORNIA,  USA DUBLIN,  IRELAND
www.usenix.org/srecon15 www.usenix.org/srecon15europe

Following 2014’s inaugural sold-out conference, SREcon has expanded


to two venues for 2015.

If you already work in an SRE environment—or want to learn how it’s


being used by many of the largest companies today—take advantage of
this rare opportunity to meet with other engineers and discuss tricks of
the trade.

Register today at www.usenix.org

LJ251-March2015.indd 67 2/19/15 9:23 AM


srecon15_lj.indd 1 2/17/15 11:03 AM
FEATURE Initializing and Managing Services in Linux: Past, Present and Future

INITIALIZING
AND MANAGING
SERVICES
IN LINUX:
PAST,
PRESENT
AND
FUTURE
systemd is the new init system
used by many of the top Linux distributions,
but do you know the history behind it
and how we got here?
Learn about the history of init systems
in Linux and their UNIX legacy.
Gain a better perspective about how Linux
manages services and other support processes.
Jonas Gorauskas

68 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 68 2/19/15 9:23 AM


O
ne of the most crucial pieces be considered a session manager,
of any UNIX-like operating because it takes care of many aspects
system is the init dæmon of userspace and its processes once
process. In Linux, this process is the system is up and running.
started by the kernel, and it’s the first The call to start this process is, in
userspace process to spawn and the fact, hard-coded in the Linux kernel.
last one to die during shutdown. Download the latest kernel sources
During the history of UNIX and and look for a function called
Linux, many init systems have gained kernel_init in the file init/main.c.
popularity and then faded away. In Among the files that the Linux kernel
this article, I focus on the history will try to execute is /sbin/init. If Linux
of the init system as it relates to cannot find one of these processes, it
Linux, and I talk about the role of throws a kernel panic and halts.
init in a modern Linux system. I also The kernel gives the init process an
relate some of the history of the ID of 1 or PID 1. All other userspace
3YSTEM 6 )NIT 3YS6 SCHEME WHICH processes are forked from init, and
was the de facto standard for many therefore, PID 1 claims ancestral rights
Linux distributions for a long time. to all other userspace processes. PID 1
Then I cover a couple more modern also automatically will become the
approaches to system initialization, direct parent process of any userspace
such as Upstart and systemd. Finally, process that is orphaned.
I pay some attention to how things
work in systemd, as this seems to be A Little Bit of History
the popular choice at the moment for Now that I have set the stage for the
several of the largest distributions. article and given you a very basic
understanding of what init is and
The Role of Init does, I’d like to digress into a little bit
Init is short for initializer, and it’s of UNIX history.
both a startup manager and a session There has been a lot of diversity in
manager for Linux and other UNIXes. the initialization schemes for UNIX-
It’s a startup manager, because it like operating systems over time. Two
plays a crucial role in the startup of of the most important init schemes
Linux. It’s the process that creates or that had a historical impact on how
initializes userspace and, ultimately, different Linux distributions do things
all userspace processes. It also may ARE THE RC SCHEME USED IN THE  "3$

WWW.LINUXJOURNAL.COM / MARCH 2015 / 69

LJ251-March2015.indd 69 2/19/15 9:23 AM


FEATURE Initializing and Managing Services in Linux: Past, Present and Future

A Linux distribution implementing a SysV


scheme can be in one of many distinct
states in which a predetermined number
of processes may be running.

AND THE 3YS6 SCHEME USED IN 3UN/3 than the original.


and Solaris. Most other Linux distributions
4HE  "3$ INIT SYSTEM IS PRETTY have, historically, been adepts of
simple and monolithic. When booting, THE 3YS6 SCHEME WHICH ORIGINALLY
the kernel runs /sbin/init, which would was implemented in AT&T UNIX and
spawn a shell to run the /etc/rc script. derivative systems like Solaris.
The /etc/rc script contained commands
to check the integrity of hard System V Init
drives and mount them, start other A Linux distribution implementing
processes, and start the networking A 3YS6 SCHEME CAN BE IN ONE OF
subsystem. This scheme was contained many distinct states in which a
completely within a few scripts: predetermined number of processes
namely /etc/rc, /etc/rc.local and may be running. These states are
/etc/netstart. This scheme also had called runlevels and to get to a certain
no specific shutdown procedure. Init runlevel means that the system is in a
WOULD RECEIVE A 3)'4%2- SIGNAL AND certain operational stage.
SEND A 3)'(50 ANDOR A 3)'4%2- TO The meaning for each runlevel may
its children, and after all processes vary based on your distribution of
exited, it would drop to single-user Linux. For example, there are a few
mode and shut down. distributions (such as Ubuntu) that
Today, the systems that have use runlevel 2 to mean multi-user
inherited the rc initialization scheme graphical mode with networking
are Free-BSD, Net-BSD and the enabled. Others (like Fedora) use
Slackware Linux distribution. These runlevel 5 to mean the same thing.
modern systems have improved )N A 3YS6 ,INUX MACHINE THE KERNEL
QUITE A BIT ON THE ORIGINAL  "3$ runs /sbin/init as usual, which in turn
scheme and are much more modular will load parameters and execute

70 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 70 2/19/15 9:23 AM


directives defined in /etc/inittab. and Kill scripts execute before Start
This file defines the default runlevel scripts. The last thing to happen is
for the whole system, describes to run the /etc/rc.local script, which
what happens when Ctrl-Alt-Del is is where you can add custom system
pressed, loads keymap files, defines commands that you want to execute
which terminals to spawn gettys for, at startup.
spawns terminal login processes, runs ! SYSTEM THAT USES THE 3YS6 SCHEME
the /etc/init.d/rcS script, and it also usually comes with the service
influences the order of execution of program used to manage the services
other runlevel scripts. while the system is running. You can
The /etc/init.d/rcS script will put check on the status of a service, or all
the system in a single-user mode in services, and start or stop a service,
order to finish probing hardware, respectively, using the service utility:
mount disks, set hostname, set up
networking and so on. Take a look Q $ service <service> status
at /etc/rcS.d/ in a Debian 7 system
for all the gory details. Next, Q $ service status -all
/sbin/init will switch itself to
the default runlevel to start all Q # service <service> start|stop
the system services. The default
runlevel value is defined in the To manage the assignment of
initdefault line of /etc/inittab. services to a particular runlevel, you
This actually translates into a call can use a tool called sysv-rc-conf ,
to the /etc/init.d/rc script with the which manages the setup of all links
parameter of 2 for the runlevel value. in the respective rc directories. You
The rc script will then execute all of the also can switch the runlevel of the
+
FOR +ILL AND 3
FOR 3TART SCRIPTS system at any time when you use the
in the /etc/rc2.d/ directory. These are command telinit as a privileged
actually links to the real scripts in user. For example, telinit 6 will
/etc/init.d/. The names of the links REBOOT A 3YS6 SYSTEM
follow the format S##<service-name> 4HE 3YS6 SCHEME STILL IS IN USE
or K##<service-name> , where the today in Debian 7 (Wheezy) systems.
## token is the two-digit number used However, the Debian developers will
to determine the order in which the be changing the init system in version
script should run. Order is alphabetical, 8 to systemd. I cover systemd in more

WWW.LINUXJOURNAL.COM / MARCH 2015 / 71

LJ251-March2015.indd 71 2/19/15 9:23 AM


FEATURE Initializing and Managing Services in Linux: Past, Present and Future

The SysV scheme has been great,


but it started to show its age around
the time when Linux on the desktop
gained a little more momentum.
detail below, but first, let’s look at shutting down services prior to
why we need a new init system. shutdown. As a result, the design
was strictly synchronous, blocking
The Problem with System V Init future tasks until the current one
4HE 3YS6 SCHEME HAS BEEN GREAT had completed.
but it started to show its age around This left the system unable to
the time when Linux on the desktop handle various events that were not
gained a little more momentum. related to the startup or shutdown
7HEN THE 3YS6 SCHEME ORIGINALLY of the system. Things that we
was designed, computers where take for granted today were really
NOTHING LIKE THEY ARE TODAY 3YS6 cumbersome to handle elegantly
was not designed to handle certain DURING THE HEYDAY OF 3YS6 INIT
things well:
Q There was no real process
Q USB devices. supervision—for example, dæmons
were not automatically restarted
Q %XTERNAL STORAGE VOLUMES when they crashed.

Q Bluetooth devices. Q There was no real dependency


checking. The order of script
Q The cloud. naming determined the order in
which they were loaded.
4HE 3YS6 SCHEME WAS DESIGNED
for a world that was static and slow Q The addition or removal of USB
moving. This init scheme originally drives and other portable storage/
was responsible only for bringing network devices while the machine
the system into a normal running was running was cumbersome and
state after power on or gracefully OFTENTIMES REQUIRED A REBOOT

72 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 72 2/19/15 9:23 AM


Q There were no facilities to spawns a process. As such, system
discover and scan for new storage initialization can be expressed as a
devices without locking the consecutive set of “spawn process X
system, especially when a disk when event Y occurs” rules.
might not even power on until it *UST LIKE IN THE 3YS6 SCHEME
was scanned. the Linux kernel gives control
to Upstart when it executes the
Q There were no facilities to load Upstart implementation of /sbin/init.
firmware for a device, which may At this point, things may work a
have needed to occur after it was little differently depending on your
detected but before it was usable. distribution of Linux. For Red Hat
%NTERPRISE ,INUX 2(%,  USERS
Inevitably, around the 2005/2006 you’ll still find a file at /etc/inittab,
time frame, several alternative but the sole function of this file is
efforts tried to fix all the issues with to set the default runlevel for the
THE 3YS6 SCHEME "UT THE EFFORT THAT system. If your distribution is one of
looked most promising during that the Ubuntu derivatives, /etc/inittab
time was the Upstart init project doesn’t even exist anymore, and the
sponsored by Canonical. default runlevel is set in a file called
/etc/init/rc-sysinit.conf instead.
Upstart The Upstart version of /sbin/init will
To be sure, Upstart init doesn’t share emit a single event called startup,
ANY CODE WITH THE 3YS6 INIT SCHEME which triggers the rest of the system
but it’s rather a superset of it, initialization. There are a few jobs
providing a good degree of backward- that specify the startup event as their
compatibility. The main departure start condition, the most notable of
FROM THE TRADITIONAL 3YS6 WAY OF DOING which is mountall , which mounts all
things is that Upstart implements an filesystems. The mountall job then
event-driven model that allows it to triggers various other events related
respond to milestones asynchronously to disk and filesystem initialization.
as they are reached. Upstart also These events, in turn, trigger the
implements the concept of jobs, udev kernel device manager to start,
which are described by the files under and it emits the event that starts the
ETCINIT
CONF AND WHOSE PURPOSE networking subsystem.
is to execute a script section that This is when one of the most critical

WWW.LINUXJOURNAL.COM / MARCH 2015 / 73

LJ251-March2015.indd 73 2/19/15 9:23 AM


FEATURE Initializing and Managing Services in Linux: Past, Present and Future

jobs is triggered by Upstart. This job is The Upstart scheme has been used
called rc-sysinit , which has a start in popular distributions of Linux, such
dependency on the filesystem and AS &EDORA FROM VERSIONS  UP TO 
network-up events. The role of this THE 2(%,  SERIES AND 5BUNTU SINCE
job is to bring the system to its default version 6.10 to present. But for all
runlevel. It executes the command the flexibility that Upstart init brings
telinit <runlevel> to achieve this. to Linux, it still falls short in a few
The telinit command then emits fundamental ways:
the runlevel event, which causes many
other jobs to start. This includes the Q It ignores the system state between
/etc/init/rc.conf job, which implements events. For instance, a system has
A COMPATIBILITY LAYER FOR THE 3YS6 INIT a power cord plugged in, then the
scheme. It executes /etc/init.d/rc system runs on AC power for a
<runlevel> and determines if a while, and then the user unplugs the
/etc/rc#.d/ directory exists for the current power cord. Upstart focuses on each
runlevel, executing all scripts in it. event above as a single discrete and
In Upstart-based systems, such unrelated unit, instead of tracking
AS 5BUNTU AND 2(%,  YOU CAN the chain of events as a whole.
use the tools sysv-rc-conf or
chkconfig , respectively, to manage Q The event-driven nature of the
the runlevel of different services. system turns the dependency chain
You also can manage jobs via the on its head. Instead of doing the
initctl utility. You can list all jobs absolute minimum amount of
and their respective start and stop work needed to get the system to
events with the command initctl a working state, when an event
show-config . You also can check is triggered, it executes all jobs
on job status, list available jobs and that could possibly follow it. For
start/stop jobs with the following example, just because networking
commands, respectively: has started, it doesn’t mean that
NFS also should start. As a matter
Q $ initctl status <job> of fact, the opposite is the correct
order of things: when a user
Q $ initctl list REQUESTS ACCESS TO AN .&3 SHARE
the system should validate that
Q # initctl start|stop <job> networking is also up and running.

74 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 74 2/19/15 9:23 AM


The main design goals of this init scheme
are, according to Lennart Poettering,
lead developer of systemd, “to start less,
and to start more in parallel”.

Q The dependency chain is still you run as much as possible at the


present. Although many more things same time.
happen in parallel in Upstart, the To accomplish these goals, systemd
user has to port the original script aims to act against two major trouble
SEQUENCE FROM 3YS6 INIT TO A SET spots of previous init schemes: the
of event trigger action rules in the shell and parallelism. The main

CONF FILES IN ETCINIT &URTHERMORE executable for systemd, /lib/systemd/
because of the spanning tree systemd, performs all calls that
structure of the event system, it is originally were present in scripts, thus
a real nightmare to figure out why eliminating the need to spawn a shell
something happened and what environment. What about the call to
event triggered it. /sbin/init that’s hard-coded in the Linux
kernel? It’s still there in the form of a
There is another init scheme symbolic link to /lib/systemd/systemd.
whose purpose is to address the To address parallelism, you need to
issues listed above. remove the dependency chain between
the various services or at least make it
systemd a secondary concern. If you look at the
systemd is the latest milestone on the problem at its most fundamental level,
road to init system nirvana. The main the dependency between the various
design goals of this init scheme are, services boils down to one thing: having
according to Lennart Poettering, lead a socket available for the processes
developer of systemd, “to start less, to communicate among themselves.
and to start more in parallel”. What systemd creates all sockets first and
that means is that you execute only then spawns all processes in parallel.
that which is absolutely necessary to For example, services that need to write
get the system to a running state, and to the system log need to wait for the

WWW.LINUXJOURNAL.COM / MARCH 2015 / 75

LJ251-March2015.indd 75 2/19/15 9:23 AM


FEATURE Initializing and Managing Services in Linux: Past, Present and Future

/dev/log socket to become available,


but as soon as it is available, these [Service]
services can start. Therefore, if systemd ExecStart=/usr/bin/sshd -D
creates the socket /dev/log first, then ExecReload=/bin/kill -HUP $MAINPID
that’s one less dependency that blocks KillMode=process
OTHER SERVICES %VEN IF THERE IS NOTHING Restart=always
to receive messages at the other end of
the socket, this strategy still works. The [Install]
kernel itself will manage a buffer for WantedBy=multi-user.target
the socket, and as soon as the receiving
service starts, it will flush the buffer This format is really simple and
and handle all the messages. The ideas really portable across several different
above are not new or revolutionary. distributions. There are other types
They have been tried before in projects of unit files that describe a system,
like the xinetd superserver and the and they are socket, device, mount,
launchd init scheme used in OS X. automount, swap, target, path, timer,
systemd does introduce the new snapshot, slice and scope. Going into
concepts of units and targets. A target all of them in detail is beyond the
is analogous to a runlevel in previous scope of this article; however, I want to
schemes and is composed of several mention one thing: target is a special
units. systemd will execute units to type of unit file that glues the other
reach a target. The instructions for each types together into a cohesive whole.
unit reside in the /lib/systemd/system/ For example, here are the contents of
directory. These files use a declarative basic.target from Arch Linux:
format that looks like a Windows INI
file. The most common type of these [Unit]

units is the service unit, which is used Description=Basic System

to start a service. The sshd.service file Documentation=man:systemd.special(7)

from Arch Linux looks like this: Requires=sysinit.target

Wants=sockets.target timers.target paths.target

[Unit] ´slices.target

Description=OpenSSH Daemon After=sysinit.target sockets.target timers.target

Wants=sshdgenkeys.service ´paths.target slices.target

After=sshdgenkeys.service JobTimeoutSec=15min

After=network.target JobTimeoutAction=poweroff-force

76 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 76 2/19/15 9:23 AM


You can follow the chain of directives Restart or RestartSec
dependencies if you look at what in your unit files. This feature allows
BASICTARGET REQUIRES AND WANTS systemd to take the role of process
Those are actual unit files in the supervisor as well.
same /lib/systemd/system/ directory. systemd refers to the init
The Requires and Wants directives dæmon executable itself, namely
above are how systemd defines /lib/systemd/systemd, but it also
the dependency chain among the refers to the set of utilities and
units. The Requires directive programs used to manage the system
DENOTES A HARD REQUIREMENT and services. Chief among these
and Wants denotes an optional utilities is the systemctl program
REQUIREMENT !LSO KEEP IN MIND THAT that’s used to manage services.
Requires and Wants don’t imply You can use it to enable, start and
order. If the After directive isn’t disable services, find the status of a
specified, systemd will start the given service and also list all loaded
units in parallel. units. For example:
Timer units are also really
interesting. They are unit files Q # systemctl enable sshd
that contain a [Timer] section
and define how the TimeDateD Q # systemctl start sshd
subsystem of systemd will activate
a future event. In these timer units, Q # systemctl stop sshd
you can create two types of timers:
one that will activate after a time Q # systemctl status sshd
period based on a variable starting
point, such as the systems boot, Q # systemctl list-units
and another that activates at fixed
intervals like a cron job. As a matter Some Linux distributions, like
of fact, timer units are an alternative 2(%,  AND #ENT/3  PROVIDE A
to cron jobs. compatibility layer that translates
One last thing to mention about 3YS6 AND 5PSTART COMMANDS INTO
systemd unit files is that they systemd commands. If you issue
provide the means to describe easily the command service sshd
what to do when a service crashes. status in CentOS 7, you will get
You can do that by using the the following output:

WWW.LINUXJOURNAL.COM / MARCH 2015 / 77

LJ251-March2015.indd 77 2/19/15 9:23 AM


FEATURE Initializing and Managing Services in Linux: Past, Present and Future

Redirecting to /bin/systemctl status sshd.service Q Display log since last boot:


sshd.service - OpenSSH server daemon # journalctl -b
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled)

Active: active (running) since Mon 2014-12-08 02:01:53 PST; Q Display errors from last boot:
´12h ago # journalctl -b -p err
Process: 915 ExecStartPre=/usr/sbin/sshd-keygen (code=exited,

´status=0/SUCCESS) I urge you to look at the


Main PID: 937 (sshd) documentation of the different schemes
CGroup: /system.slice/sshd.service presented here to learn more.
...937 /usr/sbin/sshd -D

Controversies
Notice that first line of console From my vantage point, the future is
output above and how it indicates not 100% certain when it comes to init
THAT THE 3YS6 STYLE COMMAND WAS schemes for Linux. The clear leader, as
redirected to the systemd-style of ) WRITE THIS IN LATE  IS SYSTEMD !
command. This allows the user to lot of distributions are adopting it; the
ease into the systemd way of doing LATEST ONES ARE 2(%,  AND $EBIAN 
things while still allowing the user to However, the adoption of systemd
leverage the previous skill set. has been controversial, and these
Another really important program distributions have received a lot of
in the systemd toolbox is the strong feedback from their respective
journalctl utility. It allows you communities. Of note is the Debian
to view and manage the systemd technical committee debate that
logging subsystem called journald. occurred in the Debian mailing list and
systemd’s logfile is a binary file and a complaint by Linus Torvalds himself
using journalctl really simplifies in the Linux kernel mailing list.
the user experience. Here are some systemd is not just an init scheme.
interesting examples: It unifies everything that is related to
starting and managing system services
Q Display full log: # journalctl --all into a centralized and monolithic
whole: user login, cron jobs, network
Q Tail the log: # journalctl -f services, virtual TTY management
and so on. The use of shell scripts to
Q Filter log by executable: control system startup has the benefit
# journalctl /lib/systemd/systemd of providing flexibility, and a lot of

78 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 78 2/19/15 9:23 AM


Where every interaction matters.

break down
your innovation barriers
power your business to its full potential
When you’re presented with new opportunities, you want to focus on turning
them into successes, not whether your IT solution can support them.

Peer 1 Hosting powers your business with our wholly owned FastFiber NetworkTM,

solutions that are secure, scalable, and customized for your business.

Unsurpassed performance and reliability help build your business foundation to


be rock-solid, ready for high growth, and deliver the fast user experience your
customers expect.

Want more on cloud?


Call: 844.855.6655 | go.peer1.com/linux | Vew Cloud Webinar:

Public and Private Cloud | Managed Hosting | Dedicated Hosting | Colocation

LJ251-March2015.indd 79 2/19/15 9:23 AM


FEATURE Initializing and Managing Services in Linux: Past, Present and Future

members of the community want to influenced user choice in this space


be able to choose their favorite init over time. I hope this article will foster
scheme. This has spawned some forks further discussion, and your feedback
of systemd and even a faction of the is highly encouraged. Q
Linux community that is for completely
boycotting systemd. Check out the Jonas Gorauskas is technically a software developer by trade
site http://boycottsystemd.org. but also a generalist with background in operations. In the past
he has been one or more of the following: programmer, technical
Conclusion support analyst, technical writer, systems designer, database
The userspace initialization and administrator, amateur cook and professional curmudgeon.
management of Linux systems has Jonas is currently working at Intuit in Reno, Nevada, as part of
had a rich and diverse history. I hope the Application Operations Engineering team helping them with
that this article has given you a new operations, deployment, DevOps or anything else they can think of.
perspective for how we got to where
we are today with systemd becoming
the new standard. I have covered all Send comments or feedback via
the pros and cons of the different http://www.linuxjournal.com/contact
schemes and how those factors have or to ljeditor@linuxjournal.com.

Resources
The source code of various Linux distributions, including:

Q Debian 7 and 8

Q CentOS 6.5 and 7

Q Slackware 14

Q Fedora 20

Q Ubuntu 12.4 and 14.4

Q Arch Linux

The Web site of Lennart Poettering: http://0pointer.net/blog

The systemd Documentation: http://freedesktop.org/wiki/Software/systemd

Upstart Documentation: http://upstart.ubuntu.com/cookbook

80 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 80 2/19/15 9:23 AM


NEW!
Linux Journal
eBook Series
GEEK GUIDES FREE
Down
loa
Slow Down to Speed Up: NOW d
!
Continuous Quality Assurance in a DevOps Environment
By Bill Childers
DevOps is one of the newest and largest movements in Information
Technology in the past few years. The name DevOps is a portmanteau
of “Development” and “Operations” and is meant to denote a fusion of
these two functions in a company. Whether or not your business actually
does combine the two functions, the lessons and tools learned from the
DevOps movement and attitude can be applied throughout the entire
Information Technology space. This eBook focuses on one of the key
attributes of the DevOps movement: Quality Assurance. At any point,
you should be able to release your product, code or configuration—so
long as you continue keeping your deliverables in a deployable state. This is done by “slowing
down” to include a Quality Assurance step at each point in your workflow. The sooner you catch
an error or trouble condition and fix it, the faster you can get back on track. This will lower the
amount of rework required and keep your team’s momentum going in a forward direction,
enabling your group to move on to new projects and challenges.

Build a Private Cloud for Less Than $10,000!


By Mike Diehl
This eBook presents a compelling argument as to why you should
consider re-architecting your enterprise toward a private cloud. It
outlines some of the design considerations that you need to be
aware of before implementing your own private cloud, and it
describes using the DevCloud installer in order to install OpenStack
on an Ubuntu 14 server. Finally, this eBook will familiarize you with
the features and day-to-day operations of an OpenStack-based
private cloud architecture, all for less than $10K!

DOWNLOAD NOW AT: http://linuxjournal.com/geekguides

LJ251-March2015.indd 81 2/19/15 9:23 AM


FEATURE Infinite BusyBox with systemd

Infinite
BusyBox
with
systemd
Lightweight virtual containers
with PID 1.

Charles Fisher

82 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 82 2/19/15 9:23 AM


I
n this article, I demonstrate indicated otherwise:
a method to build one Linux
system within another using cd /home

the latest utilities within the wget http://busybox.net/downloads/binaries/latest/busybox-x86_64

systemd suite of management


tools. The guest OS container You also can get a binary copy of
design focuses upon BusyBox the Dropbear SSH server and client
and Dropbear for the userspace from this location:
system utilities, but I also work
through methods for running more wget http://landley.net/aboriginal/downloads/
general application software so the ´binaries/extras/dropbearmulti-x86_64
containers are actually useful.
This tutorial was developed on For this article, I used the
Oracle Linux 7, and it likely will run following versions:
unchanged on its common brethren
(Red Hat, CentOS, Scientific Linux), Q BusyBox v1.21.1.
and from here forward, I refer to this
PLATFORM SIMPLY AS 6 3LIGHT CHANGES Q $ROPBEAR 33( MULTI PURPOSE V
may be necessary on other systemd
PLATFORMS SUCH AS 353% $EBIAN OR These are static binaries that do not
5BUNTU  /RACLES 6 RUNS ONLY ON THE link against shared objects—nothing
X? PLATFORM SO THATS THIS ARTICLES ELSE IS REQUIRED TO RUN THEM AND THEY
primary focus. are ideal for building a new UNIX-ish
ENVIRONMENT QUICKLY
Required Utilities
Red Hat saw fit to remove the Build a chroot
long-included BusyBox binary from The chroot system call and the
ITS 6 DISTRIBUTION BUT THIS EASILY associated shell utility allow an
is remedied by downloading the arbitrary subdirectory somewhere
latest binary directly from the on the system to be declared as
project’s Web site. Since the /home the root for all child processes.
filesystem gets a large amount of The commands below populate the
space by default when installing “chroot jail”, then lock you in. Note
6 LETS PUT IT THERE FOR NOW 2UN that the call to chroot needs your
the commands below as root until CHANGE TO THE 3(%,, ENVIRONMENT

WWW.LINUXJOURNAL.COM / MARCH 2015 / 83

LJ251-March2015.indd 83 2/19/15 9:23 AM


FEATURE Infinite BusyBox with systemd

BusyBox changes its behavior depending upon


how it is called—it bundles a whole system of
utility programs into one convenient package.

variable below, as you don’t have system of utility programs into one
bash inside the jail (and it’s likely the convenient package.
DEFAULT VALUE OF 3(%,,  Try a few additional UNIX
commands that you may know. Some
export SHELL=/bin/sh that work are vi , uname , uptime
mkdir /home/nifty and (of course) the shell that you are
mkdir /home/nifty/bin working inside. Commands that don’t
cd /home/nifty/bin work include ps , top and netstat .
cp /home/busybox-x86_64 /home/dropbearmulti-x86_64 . 4HEY FAIL BECAUSE THEY REQUIRE THE
chmod 755 busybox-x86_64 dropbearmulti-x86_64 /proc directory (which is dynamically
./busybox-x86_64 --list | awk '{print "ln -s provided by the Linux kernel)—it has
´busybox-x86_64 " $0}' | sh not been mounted within the jail.
chroot /home/nifty Note that few native utilities will
export PATH=/bin run in the chroot without moving
ls -l many dependent libraries (objects).
###(try some commands) You might try copying bash or gawk
exit into the jail, but you won’t be able to
run them (yet). In this regard, BusyBox
Take some time to explore your is ideal, as it depends upon nothing.
shell environment after you launch
your chroot above before you Build a Minimal UNIX System
exit. Notice that you have a /bin and Launch It
directory, which is populated by The systemd suite includes the
soft links that resolve to the eponymous program that runs
BusyBox binary. BusyBox changes as PID 1 on Linux. Among many
its behavior depending upon how other utilities, it also includes the
it is called—it bundles a whole nspawn program that is used to

84 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 84 2/19/15 9:23 AM


launch containers. Containers that that appear in the child container
are created by nspawn fix most of also appear on the host system, but
the problems with chroot jails. They different PIDs will be assigned between
provide /proc, /dev, /run and otherwise the parent and child.
EQUIP THE CHILD ENVIRONMENT WITH A Note that you’ll also receive the
more capable runtime. message: “The kernel auditing
Next, you are going to configure subsystem is known to be
a getty to run on the console of the incompatible with containers. Please
container that you can use to log in. make sure to turn off auditing with
Being sure that you have exited your AUDIT ON THE KERNEL COMMAND
chroot from the previous step, run the line before using systemd-nspawn.
following commands as root: Sleeping for 5s...” The audit
settings don’t seem to impact the
mkdir /home/nifty/etc BusyBox container login, but you
mkdir /home/nifty/root can adjust your kernel command
echo 'NAME="nifty busybox container"' > line in your grub configuration
´/home/nifty/etc/os-release (at least to silence the warning and
cd /home/nifty stop the delay).
ln -s bin sbin

ln -s bin usr/bin Running Dropbear SSH in


echo 'root::0:0:root:/root:/bin/sh' > Your Container
´/home/nifty/etc/passwd It’s best if you configure a non-
echo 'console::respawn:/bin/getty 38400 /dev/console' > root user of your system and forbid
´/home/nifty/etc/inittab network root logins. The reasoning
tar cf - /usr/share/zoneinfo | (cd /home/nifty; tar xvpf -) will become clear when I address
systemd-nspawn -bD /home/nifty container security.
Run all of these commands as root
After you have executed the nspawn within the container:
above, you will be presented with a
“nifty login” prompt. Log in as root cd /bin
(there is no password—yet), and try a ln -s dropbearmulti-x86_64 dropbear
few more commands. You immediately ln -s dropbearmulti-x86_64 ssh
will notice that ps and top work, and ln -s dropbearmulti-x86_64 scp
there is now a /proc. ln -s dropbearmulti-x86_64 dropbearkey
You also will notice that the processes ln -s dropbearmulti-x86_64 dropbearconvert

WWW.LINUXJOURNAL.COM / MARCH 2015 / 85

LJ251-March2015.indd 85 2/19/15 9:23 AM


FEATURE Infinite BusyBox with systemd

Above, you have established the echo root:::::::: > /etc/shadow


names that you need to call Dropbear, chmod 600 /etc/shadow
both the main client and server, echo root:x:0: > /etc/group
and the sundry key generation and passwd -a x root
management utilities.
You then generate the host keys that Note that the BusyBox passwd
will be used by this container, placing call used here generated an MD5
them in a new directory /home/nifty/ hash—there is a $1$ prefix in the
etc/dropbear (as viewed by the host): second field of /etc/shadow for root.
Additional hashing algorithms are
mkdir /etc/dropbear available from this version of the
dropbearkey -t rsa -f /etc/dropbear/dropbear_rsa_host_key passwd utility (the options -a s will
dropbearkey -t dss -f /etc/dropbear/dropbear_dss_host_key generate a $5$ SHA256 hash, and
dropbearkey -t ecdsa -f /etc/dropbear/dropbear_ecdsa_host_key -a sha512 will generate a $6$ hash).
However, Dropbear seems to be able
6ARIOUS DIRECTORIES ARE THEN CREATED to work only with $1$ hashes for now.
that you will need shortly: Finally, add a new user to the
system, and then halt the container:
mkdir -p /var/log/lastlog
mkdir /home adduser -h /home/luser -D luser
mkdir /var/run passwd -a x luser
mkdir /tmp
mkdir /var/tmp halt
chmod 01777 /tmp /var/tmp
You should see container shutdown
You then create the inittab, which will messages that are similar to a system halt.
launch syslogd and Dropbear once at When you next start your container,
startup (in addition to the existing getty it will listen on socket 2200 for
that is respawned whenever it dies): connections. If you want remote
hosts to be able to connect to your
echo ::sysinit:/bin/syslogd >> /etc/inittab container from anywhere on the
echo '::sysinit:/bin/dropbear -w -p 2200' >> /etc/inittab network, run this command as root on
the host to open a firewall port:
Next, you add a shadow file and
create a password for root: iptables -I INPUT -p tcp --dport 2200 --syn -j ACCEPT

86 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 86 2/19/15 9:23 AM


The port will be open only until  BIT LIBRARIES AVAILABLE WITH AN
you reboot. If you’d like the open argument to nspawn that establishes
port to persist across reboots, use the a bind mount:
firewall-config command from
within the X Window System (set the systemd-nspawn -bD /home/nifty --bind-ro=/usr/lib64

port on the second tab in the GUI).


In any case, run the container with Then, from within the container, run:
the previous nspawn syntax, then try
to connect from another shell within cd /
the parent host OS with the following: ln -s usr/lib64 lib64

ssh -l luser -p 2200 localhost 9OU THEN WILL FIND THAT MANY  BIT
binaries that you copy in from the
You should be able to log in to the host will run (running /bin/gawk -V
luser account under a BusyBox shell. RETURNS h'.5 !WK vˆAN ENTIRE
Oracle 12c instance is confirmed to
Executing Programs with run this way). The read-only library
Runtime Dependencies bind mount also has the benefit of
If you copy various system programs receiving security patches immediately
from /bin or /usr/bin into your when they appear on the host.
container, you immediately will notice There is a significant security
that they don’t work. They are missing problem with this, however. The root
shared objects that they need to run. user in the container has the power to
If you had previously copied the mount -o remount,rw /usr/lib64
gawk binary in from the host: and, thus, gain write access to your
host library directories. In general,
cp /bin/gawk /home/nifty/bin/ you cannot give root to a container
user that you don’t know and trust—
you would find that attempts to among other problems, these mounts
execute it fail with “gawk: not found” can be abused.
errors (on the host, there usually will You also might be tempted to
be explicit complaints about missing mount the /usr/lib directory in the
shared objects, which are not seen in same manner. The difficulty you
the container). will find is that the systemd binary
You easily can make most of the will be found under that directory

WWW.LINUXJOURNAL.COM / MARCH 2015 / 87

LJ251-March2015.indd 87 2/19/15 9:23 AM


FEATURE Infinite BusyBox with systemd

tree, and nspawn will try to execute After the configuring above,
it in preference to BusyBox init. if you manually launch the inetd
%NABLING  BIT RUNTIME SUPPORT contained in BusyBox, you will be
likely will involve more directory able to telnet to port 12323. Note
and mounting gymnastics than was THAT THE 6 PLATFORM DOES NOT
REQUIRED FOR USRLIB include a telnet client by default,
And now, I’m going off on a tangent. so you either can install it with yum
or use the BusyBox client (which
systemd Service Files the example below will do). Unless
You will need to call on the host PID you open up port 12323 on your
1 (systemd) directly to launch your firewall, you will have to telnet
container in an automated manner, to localhost.
potentially at boot. To do this, you Make sure any inetd that you
need to create a service file. started is shut down before
Because there is a dearth of clear proceeding to create an inetd
discussion on moving inittab and service file below:
service functions into systemd, I’ll
cover all the basic uses before creating echo '[Unit]
a service file for the container. Description=busybox inetd
Start by configuring a telnet server. #After=network-online.target
The telnet protocol is not secure, as Wants=network-online.target
it transmits passwords in clear text.
Don’t practice these examples on a [Service]
production server or with sensitive #ExecStartPre=
information or accounts. #ExecStopPost=
Classical telnetd is launched by #Environment=GZIP=-9
the inetd superserver, both of which
are implemented by BusyBox. Let’s #OPTION 1
configure inetd for telnet on port ExecStart=/home/nifty/bin/inetd -f
12323. Run the following as root Type=simple
on the host: KillMode=process

echo '12323 stream tcp nowait root #OPTION 2


´/home/nifty/bin/telnetd telnetd -i -l #ExecStart=/home/nifty/bin/inetd
/home/nifty/bin/login' >> /etc/inetd.conf #Type=forking

88 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 88 2/19/15 9:23 AM


#Restart=always S
#User=root Kernel 3.10.0-123.9.3.el7.x86_64 on an x86_64
#Group=root localhost.localdomain login: jdoe
Password:
[Install]
WantedBy=multi-user.target' > Checking the status again, you see
´/etc/systemd/system/inetd.service information about the connection and
the session activity:
systemctl start inetd.service
[root@localhost ~]# systemctl status inetd.service

After starting the inet service inetd.service - busybox inetd

above, you can check the status Loaded: loaded (/etc/systemd/system/inetd.service; disabled)

of the dæmon: Active: active (running) since Sun 2014-11-16 12:34:04 CST;

´7min ago

[root@localhost ~]# systemctl status inetd.service Main PID: 3927 (inetd)

inetd.service - busybox inetd CGroup: /system.slice/inetd.service

Loaded: loaded (/etc/systemd/system/inetd.service; disabled) ´3927 /home/nifty/bin/inetd -f

Active: active (running) since Sun 2014-11-16 12:21:29 CST; ´4076 telnetd -i -l /home/nifty/bin/login

´28s ago ´4077 -bash

Main PID: 3375 (inetd)

CGroup: /system.slice/inetd.service You can learn more about


´3375 /home/nifty/bin/inetd -f systemd service files with the man
5 systemd.service command.
Nov 16 12:21:29 localhost.localdomain systemd[1]: Started There is an important point to make
´busybox inetd. here—you have started inetd with the
Try opening a telnet session from a different console: “-f Run in foreground” option. This
is not how inetd normally is started—
/home/nifty/bin/telnet localhost 12323 this option is commonly used for
debugging activity. However, if you
You should be presented with a were starting inetd with a classical
login prompt: inittab entry, -f would be useful in
conjunction with “respawn”. Without
Entering character mode -f , inetd immediately will fork into
Escape character is '^]'. the background; attempting to

WWW.LINUXJOURNAL.COM / MARCH 2015 / 89

LJ251-March2015.indd 89 2/19/15 9:23 AM


FEATURE Infinite BusyBox with systemd

respawn forking dæmons will launch thus, safer.


them repeatedly. With -f , you can You can learn more about the
configure init to relaunch inetd should KillMode option with the man
it die. 5 systemd.kill command.
Another important point is stopping Note also that the systemctl
the service. With a foreground status output included the word
dæmon and the KillMode=process “disabled”. This indicates that
setting in the service file, the child the service will not be started at
telnetd services are not killed when boot. Pass the enable keyword to
the service is stopped. This is not systemctl for the service to set it
the normal, default behavior for to launch at boot (the disable
a systemd service, where all the keyword will undo this).
children will be killed. Make some note of the
To see this mass kill behavior, commented options above. You
comment out the OPTION 1 settings may set environment variables for
in the service file (/etc/systemd/ your service (here suggesting a
system/inetd.service), and enable COMPRESSION QUALITY SPECIFY A NON ROOT
the default settings in OPTION 2 . user/group and commands to be
Then execute: executed before the service starts or
after it is halted. These capabilities
systemctl stop inetd.service are beyond the direct features
systemctl daemon-reload offered by the classical inittab.
systemctl start inetd.service Of course, systemd is capable of
spawning telnet servers directly,
Launch another telnet session, then allowing you to dispense with inetd
stop the service. When you do, your altogether. Run the following as root
telnet sessions will all be cut with on the host to configure systemd for
“Connection closed by foreign host.” BusyBox telnetd:
In short, the default behavior of
systemd is to kill all the children of a systemctl stop inetd.service

service when a parent dies.


The KillMode=process setting can echo '[Unit]

be used with the forking version of Description=mytelnet

inetd, but the “-f Run in foreground”


in the first option is more specific and, [Socket]

90 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 90 2/19/15 9:23 AM


ListenStream=12323 to the telnet server indicates
Accept=yes that systemd should not pay
attention to any stats return codes
[Install] from the process.
WantedBy=sockets.target' >

´/etc/systemd/system/mytelnet.socket Q In the client telnet sessions, the


command cat /proc/self/
echo '[Unit] cgroup will return detailed
Description=mytelnet connection information for the
IP addresses involved.
[Service]

ExecStart=-/home/nifty/bin/telnetd telnetd -i -l At this point, I have returned from


´/home/nifty/bin/login my long-winded tangent, so now let’s
StandardInput=socket' > build a service file for the container.
´/etc/systemd/system/mytelnet@.service Run the following as root on the host:

systemctl start mytelnet.socket echo '[Unit]

Description=nifty container

Some notes about inetd-style services:


[Service]

Q The socket is started, rather than ExecStart=/usr/bin/systemd-nspawn -bD /home/nifty

the service, when inetd services are KillMode=process' > /etc/systemd/system/nifty.service

launched. Similarly, they are enabled


to set them to launch at boot. Be sure that you have shut down
any other instances of the nifty
Q The @ character in the service file container. You optionally can
indicates this is an “instantiated” disable the console getty by
service. They are used when a commenting/removing the first line
number of similar services are of /home/nifty/etc/inittab. Then use PID 1
launched with a single service file to launch your container directly:
(getty being the prime example—
they also work well for Oracle systemctl start nifty.service
database instances).
If you check the status of the
Q The - prefix above in the path service, you will see the same level of

WWW.LINUXJOURNAL.COM / MARCH 2015 / 91

LJ251-March2015.indd 91 2/19/15 9:23 AM


FEATURE Infinite BusyBox with systemd

information that you previously saw [root@localhost ~]# size /home/busybox-x86_64

on the console: text data bss dec hex filename

942326 29772 19440 991538 f2132 /home/busybox-x86_64

[root@localhost ~]# systemctl status nifty.service

nifty.service - nifty container If you want to conserve the memory


Loaded: loaded (/etc/systemd/system/nifty.service; static) used by BusyBox, one way would be
Active: active (running) since Sun 2014-11-16 14:06:21 CST; to create a common /cbin that you
´31s ago attach to all containers as a read-only
Main PID: 5881 (systemd-nspawn) bind mount (as you did previously
CGroup: /system.slice/nifty.service WITH LIB AND RESET ALL THE LINKS IN
´5881 /usr/bin/systemd-nspawn -bD /home/nifty /bin to the new location. The root user
could do this:
Nov 16 14:06:21 localhost.localdomain systemd[1]: Starting

´nifty container... systemctl stop nifty.service

Nov 16 14:06:21 localhost.localdomain systemd[1]: Started

´nifty container. mkdir /home/cbin

Nov 16 14:06:26 localhost.localdomain systemd-nspawn[5881]: mv /home/nifty/bin/busybox-x86_64 /home/cbin

´Spawning namespace container on /home/nifty mv /home/nifty/bin/dropbearmulti-x86_64 /home/cbin

´(console is /dev/pts/4). cd /

Nov 16 14:06:26 localhost.localdomain systemd-nspawn[5881]: ln -s home/cbin cbin

´Init process in the container running as PID 5883. cd /home/nifty/bin

for x in *; do if [ -h "$x" ]; then rm -f "$x"; fi; done

Memory and Disk Consumption /cbin/busybox-x86_64 --list | awk '{print "ln -s

BusyBox is a big program, and if you ´/cbin/busybox-x86_64 " $0}' | sh

are running several containers that ln -s /cbin/dropbearmulti-x86_64 dropbear

each have their own copy, you will ln -s /cbin/dropbearmulti-x86_64 ssh

waste both memory and disk space. ln -s /cbin/dropbearmulti-x86_64 scp

It is possible to share the “text” ln -s /cbin/dropbearmulti-x86_64 dropbearkey

segment of the BusyBox memory ln -s /cbin/dropbearmulti-x86_64 dropbearconvert

usage between all running programs,


but only if they are running on the You also could arrange to bind-
same inode, from the same filesystem. mount the zoneinfo directory,
The text segment is the read-only, saving a little more disk space in
compiled code of a program, and you the container (and giving the
can see the size like this: container patches for time zone

92 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 92 2/19/15 9:23 AM


It might interesting to launch tens, hundreds,
or even thousands of containers at once.

data in the bargain): Infinite BusyBox


It might interesting to launch tens,
cd /home/nifty/usr/share hundreds, or even thousands of
rm -rf zoneinfo containers at once. You could launch
the clones by making copies of the
Then the service file is modified to /home/nifty directory, then adjusting
bind /cbin and /usr/share/zoneinfo the systemd service file. To simplify,
(note the altered syntax for sharing you will place your new containers
/cbin below, when the paths differ in /home/nifty1, /home/nifty2,
between host and container): /home/nifty3 ... using integer suffixes
on the directories to differentiate them.
echo '[Unit] Please make sure that you have
Description=nifty container disabled kernel auditing to remove
the five-second delay when launching
[Service] containers. At the very least, press
ExecStart=/usr/bin/systemd-nspawn -bD /home/nifty e at the grub menu at boot time,
--bind-ro=/home/cbin:/cbin --bind-ro=/usr/share/zoneinfo and add the audit=0 to your kernel
KillMode=process' > /etc/systemd/system/nifty.service command line for a one-time boot.
I’m going to return to the subject of
systemctl daemon-reload systemd “instantiated services” that I
touched upon with the telnetd service
systemctl start nifty.service FILE THAT REPLACED INETD 4HIS TECHNIQUE
will allow you to use one service
Now any container using the file to launch all of your containers.
BusyBox binary from /cbin will share Such a service has an @ character
the same inode. All versions of the in the filename that is used to refer
BusyBox utilities running in those to a particular, differentiated instance
containers will share the same text of a service, and it allows the use
segment in memory. of the %i placeholder within the

WWW.LINUXJOURNAL.COM / MARCH 2015 / 93

LJ251-March2015.indd 93 2/19/15 9:23 AM


FEATURE Infinite BusyBox with systemd

service file for variable expansion. make a thousand of them:


Run the following on the host as
root to place your service file for cd /home

instantiated containers: for x in $(seq 1 999)

do

echo '[Unit] mkdir "nifty${x}"

Description=nifty container # %i (cd nifty; tar cf - .) | (cd "nifty${x}"; tar xpf -)

sed "s/2200/$((x+2200))/" < nifty/etc/inittab >

[Service] ´nifty${x}/etc/inittab

ExecStart=/usr/bin/systemd-nspawn -bD /home/nifty%i systemctl start nifty@${x}.service

´--bind-ro=/home/cbin:/cbin --bind-ro=/usr/share/zoneinfo done

KillMode=process' > /etc/systemd/system/nifty@.service

As you can see below, this test


The %i above first adjusts the launches all containers:
description, then adjusts the launch
directory for the nspawn . The content $ ssh -l luser -p 3199 localhost

that will replace the %i is specified on The authenticity of host '[localhost]:3199 ([::1]:3199)'

the systemctl command line. ´can't be established.

To test this, make a directory ECDSA key fingerprint is 07:26:15:75:7d:15:56:d2:ab:9e:

called /home/niftyslick. The service ´14:8a:ac:1b:32:8c.

file doesn’t limit you to numeric Are you sure you want to continue connecting (yes/no)? yes

suffixes. You will adjust the SSH Warning: Permanently added '[localhost]:3199' (ECDSA)

port after the copy. Run this as root ´to the list of known hosts.

on the host: luser@localhost's password:

~ $ sh --help

cd /home BusyBox v1.21.1 (2013-07-08 11:34:59 CDT) multi-call binary.

mkdir niftyslick

(cd nifty; tar cf - .) | (cd niftyslick; tar xpf -) Usage: sh [-/+OPTIONS] [-/+o OPT]... [-c 'SCRIPT'

sed "s/2200/2100/" < nifty/etc/inittab > niftyslick/etc/inittab ´[ARG0 [ARGS]] / FILE [ARGS]]

systemctl start nifty@slick.service Unix shell interpreter

Bearing this pattern in mind, let’s ~ $ cat /proc/self/cgroup

create a script to produce these 10:hugetlb:/

CONTAINERS IN MASSIVE QUANTITIES ,ETS 9:perf_event:/

94 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 94 2/19/15 9:23 AM


8:blkio:/ ´2882 /bin/dropbear -w -p 2210

7:net_cls:/

6:freezer:/ Nov 18 23:01:21 localhost.localdomain systemd[1]:

5:devices:/ ´Starting Container nifty10.

4:memory:/ Nov 18 23:01:21 localhost.localdomain systemd[1]:

3:cpuacct,cpu:/ ´Started Container nifty10.

2:cpuset:/

1:name=systemd:/machine.slice/machine-nifty999.scope The raw number of containers that


you can launch with this approach
The output of systemctl will list each is more directly impacted by kernel
of your containers: limits than general disk and memory
resources. Launching the containers
# systemctl above used no swap on a small system
... with 2GB of RAM.
machine-nifty1.scope loaded active running Container nifty1 After you have investigated a few
machine-nifty10.scope loaded active running Container nifty10 of the containers and their listening
machine-nifty100.scope loaded active running Container nifty100 ports, the easiest and cleanest way to
machine-nifty101.scope loaded active running Container nifty101 get all of your containers shut down is
machine-nifty102.scope loaded active running Container nifty102 likely a reboot.
...

Container Security
More detail is available with A number of concerns are raised with
systemctl status : these features:
1) Since BusyBox and Dropbear
machine-nifty10.scope - Container nifty10 were not installed with the RPM host
Loaded: loaded (/run/systemd/system/machine-nifty10.scope; package tools, updates to them will
´static) have to be loaded manually. It will
Drop-In: /run/systemd/system/machine-nifty10.scope.d be important to check from time to
´90-Description.conf, 90-Slice.conf, time if new versions are available
´90-TimeoutStopUSec.conf and if any security flaws have been
Active: active (running) since Tue 2014-11-18 23:01:21 CST; discovered. If it is necessary to load
´11min ago new versions, the binaries should
CGroup: /machine.slice/machine-nifty10.scope be copied to all containers that are
´2871 init potentially used, which should then
´2880 /bin/syslogd be restarted (especially if a security

WWW.LINUXJOURNAL.COM / MARCH 2015 / 95

LJ251-March2015.indd 95 2/19/15 9:23 AM


FEATURE Infinite BusyBox with systemd

The crux is that untrusted users cannot have


the container root, any more than you would
give them full system root.

issue is involved). and testing as well as building


2) Control of the root user in the of packages, distributions and
container cannot be passed to an software involved with boot and
individual that you do not trust. systems management.
For a particular example, if the
LIBCBINZONEINFO BIND MOUNTS The crux is that untrusted users
above are used, the container root cannot have the container root, any
user can issue the command: more than you would give them full
system root. The container root will
mount -o remount,rw /usr/lib64 have the CAP_SYS_ADMIN privilege,
which allows full control of the
at which point the container root system. If you want to isolate
will have full write privileges on your non-root users further, the container
 BIT LIBRARIES CONTAINER BIN OR ZONEINFO environment does limit non-root
The systemd-nspawn man page goes users’ visibility into host activities, as
even further, with the warning: they cannot see the full process table.
3) Note that the BusyBox su and
Note that even though these passwd utilities above do not work
security precautions are taken when installed in the manner outlined
systemd-nspawn is not suitable here. They lack the appropriate
for secure container setups. Many filesystem permissions. To fix this,
of the security features may be chmod u+s busybox-x86_64
circumvented and are hence could be executed, but this is also
primarily useful to avoid accidental distasteful from a security perspective.
changes to the host system from Removing the links and copying the
the container. The intended use BusyBox binary to su and passwd
of this program is debugging before applying the setuid privilege

96 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 96 2/19/15 9:23 AM


might be better, but only slightly. It 6 INIT IS NOT ABLE TO NSPAWN AND HAS
would be best if su was unavailable far less control over processes running
and another mechanism was found for on a system. The features delivered by
password changes. systemd surely justify the inconvenience
 4HE -w argument to the Dropbear of change in many situations.
SSH server above prevents root logins Toward the second point, much
from the network. It is somewhat thought was placed into the adoption
distasteful, from a security perspective, of the architecture of systemd
to relax this limitation. The net effect by skilled designers from diverse
is that root is locked out of active use organizations. Those most critical
in the container when -w is forced, of the new environment should
and su/passwd do not have setuid. If acknowledge the technical success
it is at all possible to live with such an of systemd as it is adopted by the
arrangement for your container, try to majority of the Linux community.
do so, as the security is much improved. In any case, the next decade will
see popular Linux server distributions
systemd Controversy EQUIPPED WITH SYSTEMD AND COMPETENT
There is a high degree of hostility administrators will not have the option
toward systemd from users of of ignoring it. It is unfortunate that the
Linux. This hostility is divided into introduction of systemd did not include
two main complaints: more guidance for the user community,
but the new features are compelling
Q The classic inittab from UNIX and should not be overlooked. Q
3YSTEM 6 SHOULD NOT BE CHANGED
because it is well understood. Charles Fisher has an electrical engineering degree from
the University of Iowa and works as a systems and database
Q Increasing features are bundled administrator for a Fortune 500 mining and manufacturing
into systemd that bring corporation. He has previously published both journal articles
dangerous complexity to a and technical manuals on Linux for UnixWorld and other
critical system process. McGraw-Hill publications.

Toward the first point, nostalgia for


legacy systems is not always misguided, Send comments or feedback via
but it cannot be allowed to hinder http://www.linuxjournal.com/contact
progress unreasonably. A classic System or to ljeditor@linuxjournal.com.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 97

LJ251-March2015.indd 97 2/19/15 9:23 AM


KNOWLEDGE HUB

WEBCASTS
Learn the 5 Critical Success Factors to Accelerate
IT Service Delivery in a Cloud-Enabled Data Center
Today's organizations face an unparalleled rate of change. Cloud-enabled data centers are increasingly seen as a way to accelerate
IT service delivery and increase utilization of resources while reducing operating expenses. Building a cloud starts with virtualizing
your IT environment, but an end-to-end cloud orchestration solution is key to optimizing the cloud to drive real productivity gains.

> http://lnxjr.nl/IBM5factors

Modernizing SAP Environments with Minimum


Risk—a Path to Big Data
Sponsor: SAP | Topic: Big Data
)S THE DATA EXPLOSION IN TODAYS WORLD A LIABILITY OR A COMPETITIVE ADVANTAGE FOR YOUR BUSINESS %XPLOITING MASSIVE AMOUNTS
of data to make sound business decisions is a business imperative for success and a high priority for many firms. With rapid
advances in x86 processing power and storage, enterprise application and database workloads are increasingly being moved
from UNIX to Linux as part of IT modernization efforts. Modernizing application environments has numerous TCO and ROI
benefits but the transformation needs to be managed carefully and performed with minimal downtime. Join this webinar to
HEAR FROM TOP )$# ANALYST 2ICHARD 6ILLARS ABOUT THE PATH YOU CAN START TAKING NOW TO ENABLE YOUR ORGANIZATION TO GET THE
benefits of turning data into actionable insights with exciting x86 technology.

> http://lnxjr.nl/modsap

WHITE PAPERS
White Paper: JBoss Enterprise Application
Platform for OpenShift Enterprise
Sponsor: DLT Solutions
2ED (ATSš *"OSS %NTERPRISE !PPLICATION 0LATFORM FOR /PEN3HIFT %NTERPRISE OFFERING PROVIDES )4 ORGANIZATIONS WITH A SIMPLE AND
STRAIGHTFORWARD WAY TO DEPLOY AND MANAGE *AVA APPLICATIONS 4HIS OPTIONAL /PEN3HIFT %NTERPRISE COMPONENT FURTHER EXTENDS
THE DEVELOPER AND MANAGEABILITY BENEFITS INHERENT IN *"OSS %NTERPRISE !PPLICATION 0LATFORM FOR ON PREMISE CLOUD ENVIRONMENTS

5NLIKE OTHER MULTI PRODUCT OFFERINGS THIS IS NOT A BUNDLING OF TWO SEPARATE PRODUCTS *"OSS %NTERPRISE -IDDLEWARE HAS BEEN
HOSTED ON THE /PEN3HIFT PUBLIC OFFERING FOR MORE THAN  MONTHS !ND MANY CAPABILITIES AND FEATURES OF *"OSS %NTERPRISE
Application Platform 6 and JBoss Developer Studio 5 (which is also included in this offering) are based upon that experience.

This real-world understanding of how application servers operate and function in cloud environments is now available in this
SINGLE ON PREMISE OFFERING *"OSS %NTERPRISE !PPLICATION 0LATFORM FOR /PEN3HIFT %NTERPRISE FOR ENTERPRISES LOOKING FOR CLOUD
benefits within their own datacenters.

> http://lnxjr.nl/jbossapp

98 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 98 2/19/15 9:23 AM


KNOWLEDGE HUB

WHITE PAPERS
Linux Management with Red Hat Satellite:
Measuring Business Impact and ROI
Sponsor: Red Hat | Topic: Linux Management

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to de-
ploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT
organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility
workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows
IN IMPORTANCE IN TERMS OF VALUE TO THE BUSINESS MANAGING ,INUX ENVIRONMENTS TO HIGH STANDARDS OF SERVICE QUALITY ˆ
AVAILABILITY SECURITY AND PERFORMANCE ˆ BECOMES AN ESSENTIAL REQUIREMENT FOR BUSINESS SUCCESS

> http://lnxjr.nl/RHS-ROI

Standardized Operating Environments


for IT Efficiency
Sponsor: Red Hat
4HE 2ED (ATš 3TANDARD /PERATING %NVIRONMENT 3/% HELPS YOU DEFINE DEPLOY AND MAINTAIN 2ED (AT %NTERPRISE ,INUXš
AND THIRD PARTY APPLICATIONS AS AN 3/% 4HE 3/% IS FULLY ALIGNED WITH YOUR REQUIREMENTS AS AN EFFECTIVE AND MANAGED
process, and fully integrated with your IT environment and processes.

Benefits of an SOE:

3/% IS A SPECIFICATION FOR A TESTED STANDARD SELECTION OF COMPUTER HARDWARE SOFTWARE AND THEIR CONFIGURATION FOR USE
ON COMPUTERS WITHIN AN ORGANIZATION 4HE MODULAR NATURE OF THE 2ED (AT 3/% LETS YOU SELECT THE MOST APPROPRIATE
solutions to address your business' IT needs.

SOE leads to:

s $RAMATICALLY REDUCED DEPLOYMENT TIME

s 3OFTWARE DEPLOYED AND CONFIGURED IN A STANDARDIZED MANNER

s 3IMPLIFIED MAINTENANCE DUE TO STANDARDIZATION

s )NCREASED STABILITY AND REDUCED SUPPORT AND MANAGEMENT COSTS

s 4HERE ARE MANY BENEFITS TO HAVING AN 3/% WITHIN LARGER ENVIRONMENTS SUCH AS

s ,ESS TOTAL COST OF OWNERSHIP 4#/ FOR THE )4 ENVIRONMENT

s -ORE EFFECTIVE SUPPORT

s &ASTER DEPLOYMENT TIMES

s 3TANDARDIZATION

> http://lnxjr.nl/RH-SOE

WWW.LINUXJOURNAL.COM / MARCH 2015 / 99

LJ251-March2015.indd 99 2/19/15 9:23 AM


EOF
Resurrecting DOC SEARLS

the Armadillo
Fifteen years after giving the world a pile of hopefully helpful
memes, Cluetrain rides again.

1999 was a crazy year for consumers”, Cluetrain said. “We are
business on the Internet, human beings and our reach exceeds
and for Linux. It was when Red your grasp. Deal with it.”
Hat went public, with a record We addressed Cluetrain to “People
VALUATION AND 6! ,INUX FOLLOWED OF %ARTHv BUILT IT AROUND  THESES
with a bigger one. Both were cases (because that worked for Martin
in point of the dot-com boom, a Luther) and called it a “manifesto”
speculative bubble inflated by huge (because that worked for Karl Marx).
expectations of what the Internet The “Cluetrain” name came from an
would mean for business. OLD 3ILICON 6ALLEY PUT DOWN h4HE
In April of that year, Chris clue train stopped there four times a
Locke, Rick Levine, David day and they never took delivery.”
Weinberger and I put up a Web site )T WAS A HIT 6OLUNTEERS TRANSLATED
called The Cluetrain Manifesto it into 13 languages. The Wall Street
(http://cluetrain.com), attempting Journal covered it on the front page
to make clear that the Internet was of its Marketplace section. Book
for everybody and everything, and offers came in. We accepted one
not just for companies looking to and wrote the book version of The
exit into wealth on Wall Street. Cluetrain Manifesto that summer. It
Our bulls-eye was marketing, came out in January 2000 and was
which spoke about users in terms a business bestseller, even though it
that were barely human. “We are could also be read for free on-line at
not seats or eyeballs or end users or the Cluetrain Web site. It went on

100 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 100 2/19/15 9:23 AM


EOF

New Clues was tuned for our time—one in which


Internet usage has been migrating from wired to
cellular connections, from the Web to apps, and
from the Net’s wide open spaces to the closed
and proprietary walled gardens of Facebook,
Twitter, Apple, Google and other feudal lords.

to be published in nine languages, degree, I suppose. But I don’t care.


and still sells at a nice clip, 15 Nor do I care that Cluetrain is often
years later. Same goes for a 10th credited with having something
anniversary edition that came to do with social media. What I
out in 2010. care about is that Cluetrain hasn’t
Today the word cluetrain, which yet succeeded at its main mission:
didn’t exist before 1999, appears in to make clear that the Internet is
thousands of books and is tweeted something more than the pipes
many times per day. One-liners we get it from, the “content” we
from its list of theses—“Markets find there, and the companies and
are conversations”, “Hyperlinks governments that would have us
SUBVERT HIERARCHYvˆARE QUOTED think they run the thing.
endlessly. Not bad for an project that 3O ON !UGUST   WHEN
had no promotion, no budget, no somebody pointed me to yet another
conferences, no bumper stickers, Cluetrain story that failed to grok
no t-shirts and no interest in what it was really about, I wrote
becoming a business or an this to the other three guys: “I feel
institution. It was just a bunch of an urge to publish something that
ideas people could put to use. says ’The Cluetrain Manifesto was
The biggest irony of Cluetrain’s not about clearing the way for social
success is that most books that media.’” David Weinberger wrote
cite it are marketing books, and back, “Anyone ready for a new
most tweets about it seem to be manifesto?” A few minutes later,
by marketing people. Is marketing he shared the first draft of one: a
better because of it? To some collection of theses, similar to the

WWW.LINUXJOURNAL.COM / MARCH 2015 / 101

LJ251-March2015.indd 101 2/19/15 9:23 AM


EOF

original in style and length. Chris Q The Internet is not content.


Locke followed with a wordless
image of a woman gleefully shooting Q The Net is not a medium.
thumbs-up images out of a machine
gun. The rest of the back-and-forth Q How did we let conversation get
was between David and me. (Chris weaponized anyway?
and Rick stayed busy with other
things.) The result was New Clues, Q Marketing still makes it harder to talk.
which went up on January 8, 2015,
at the original Cluetrain site: Q Kumbiyah sounds surprisingly good
http://cluetrain.com/newclues. in an echo chamber.
New Clues was tuned for our
time—one in which Internet usage Q The Gitmo of the Net.
has been migrating from wired
to cellular connections, from the Q Gravity’s great until it sucks us all
Web to apps, and from the Net’s into a black hole.
wide open spaces to the closed
and proprietary walled gardens of Q Privacy in an age of spies.
Facebook, Twitter, Apple, Google
and other feudal lords. “An organ- Q Privacy in an age of weasels.
by-organ body snatch of the Internet
is already well underway”, it says in Q A pocket full of homilies.
the preamble. “Make no mistake:
with a stroke of a pen, a covert Q Being together: the cause of and
handshake, or by allowing memes to solution to every problem.
drown out the cries of the afflicted
we can lose the Internet we love.” We wanted to make New Clues, and
One hundred and twenty one every piece of it, as useful, mixable
numbered clues follow, under and remixable as possible. So:
thematic subheads:
Q %VERY SUBHEAD AND EVERY CLUE HAS A
Q The Internet is us, connected. link of its own.

Q The Internet is nothing and has Q The text is released to the public
no purpose. domain with a Creative Commons

102 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 102 2/19/15 9:23 AM


EOF

We wanted to make New Clues, and every piece of


it, as useful, mixable and remixable as possible.

:ERO  5NIVERSAL ##  the great Steven Levy):


Public Domain Dedication. In https://medium.com/backchannel/
other words, no copyright at internet-under-fire-gets-new-
all. When asked for permission manifests-207a922b459e.
to republish New Clues, David
replies, “You don’t have our Q A version by Kevin Marks that
permission. Go ahead!” accepts fragmentations and
webmentions.
Q The whole thing is on GitHub,
with machine-readable versions Q A site by John Johnston that
(JSON, XML and OPML, so far). The randomly generates one clue per
GitHub folks also have offered to click: http://johnjohnston.info/
set up a way for us to maintain a oddsandends/givemeaclue.
single data file (YAML) that will
automatically create all the other Q Thousands of tweets and re-tweets
versions we need. (hashtags: #cluetrain #newclues).

The results so far (I’m writing this a Q 6OLUNTEER TRANSLATIONS INTO 'ERMAN
week after it went up) include: Italian, Italian (yes, there are two
different ones), Catalan and French
Q A listicle that Dave W iner hacked (see Resources).
together on his own software,
which he improved in the course Q Lots of great blog posts, such
of posting it. (Some people like as this one by JP Rangaswami:
the listicle version better than http://confusedofcalcutta.com/
the one-page text one. Try it 2015/01/11/new-clues-calling-on-
out: http://scripting.com.) everyone-to-be-dutiful-individuals.

Q An artful posting on Backchannel Q A Gillmor Gang devoted to


at Medium (by invitation from to the subject, with David and

WWW.LINUXJOURNAL.COM / MARCH 2015 / 103

LJ251-March2015.indd 103 2/19/15 9:23 AM


EOF

Now the question is, Will it work? Or will it be,


like so much else that gets published on the Web,
snow on the water?

myself: http://techcrunch.com/2015/
01/10/gillmor-gang-kind-of-clue.

Q A discussion page on Facebook:


https://www.facebook.com/
login.php?next=https%3A%2F%2F
www.facebook.com%2Fgroups%
2Fnewclues%2F.

Our only common design element


between Cluetrain and New Clues
is an armadillo. On Cluetrain’s index
page is the image shown in Figure 1
of a flattened armadillo in the
middle of a road, painted over with
yellow divider lines. The provenance
of the photo is unknown to us.
Chris Locke found it somewhere,
and nobody has ever stepped Figure 1. Armadillo from Cluetrain’s
forward to claim it. Index Page
The one for New Clues is shown in
Figure 2. It was posted by e. res on listicle version, Dave Winer kept the
Flickr and made useful by a Creative color but darkened it.
Commons Attribution 2.0 Generic Among the few criticisms of New
(CC BY 2.0) license. (The shot was Clues is that it’s “not so disruptive”
taken at Alki Beach in Seattle, the as Cluetrain was. For the last few
town where Linux Journal was born.) YEARS 3ILICON 6ALLEY HAS BEEN SO GAGA
For New Clues, we cropped the shot over disruption that it even has a
and made it black and white. For his conference named after it. The term

104 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 104 2/19/15 9:23 AM


comes from Clayton Christensen’s
work on disruptive innovation, which Advertiser
is defined as “a process by which a
product or service takes root initially
Index
in simple applications at the bottom Thank you as always for supporting our
of a market and then relentlessly advertisers by buying their products!
moves up market, eventually displacing
established competitors”. This can ADVERTISER URL PAGE #
apply to ideas as well as technologies.
Cases in point: free software and open Drupalize.me http://drupalize.me 108

source. Both of which, of course, %MBEDDED ,INUX HTTPEVENTSLINUXFOUNDATIONORG 


Conference events/embedded-linux-conference
informed Cluetrain and New Clues.
%MPEROR,INUX HTTPWWWEMPERORLINUXCOM 
.OW THE QUESTION IS Will it work?
Or will it be, like so much else that HPC Wallstreet http://www.flaggmgmt.com/linux 31

gets published on the Web, snow on Libre Planet 2015 https://libreplanet.org/2015/ 21

the water? Can’t say, so soon after LinuxFest Northwest http://linuxfestnorthwest.org/2015 65

it’s published. But the two publishing


Netgate http://www.netgate.com 7
dates, a decade and a half apart, came
/g2EILLY 3OFTWARE HTTPOREILLY#BB+) 
at very different times on the Web, and Architecture Conference

we did our best to leverage both. Peer 1 Hosting http://go.peer1.com/linux 79


In 1999, the Web was a collection
Silicon Mechanics http://www.siliconmechanics.com 3
of almost physical places. You put up
or built Web sites on domains with 32%CON HTTPSWWWUSENIXORGCONFERENCE 
srecon15

locations that people visited and 6AULT HTTPEVENTSLINUXFOUNDATIONORG 


browsed. Search engines might take events/vault

days or weeks to index a page. But


then, after blogs came along, with
syndication through RSS, what my son ATTENTION ADVERTISERS
Allen in 2003 described as “the Live
The Linux Journal brand’s following has
Web” began to emerge. Technorati grown to a monthly readership nearly
one million strong. Encompassing the
and other search engines for live stuff magazine, Web site, newsletters and
appeared. My October 2005 column much more, Linux Journal offers the
ideal content environment to help you
in Linux Journal was titled “The World reach your marketing objectives. For
more information, please visit
Live Web”. In it I said the Live Web was http://www.linuxjournal.com/advertising.
“about time and people, rather than

WWW.LINUXJOURNAL.COM / MARCH 2015 / 105

LJ251-March2015.indd 105 2/19/15 9:23 AM


EOF

Figure 2. New Clues Armadillo

sites and content”. This was a year week. As buzz, that’s pretty much
before Twitter and Facebook took off, what it did. But as durable food for
and search engines’ time-to-index was re-thinking what the Net is, and how
reduced nearly to zero. Sites today are we might lose it, maybe it will have
geysers of content, and the Live Web lasting effects. Sure hope so. Q
is a giant short-attention-span theater.
What shows there is hyper-social and Doc Searls is Senior Editor of Linux Journal. He is also a
temporary in the extreme, made more fellow with the Berkman Center for Internet and Society at
for sharing than for saving. Harvard University and the Center for Information Technology
This is why New Clues is a collection and Society at UC Santa Barbara.
of stuff for sharing, presented atop an
old chunk of bedrock, which is what
Cluetrain has become. If we had only Send comments or feedback via
put it on Medium, or on Facebook, http://www.linuxjournal.com/contact
it would have come and gone in a or to ljeditor@linuxjournal.com.

106 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 106 2/19/15 9:23 AM


EOF

Resources
Dot-Com Bubble: http://en.wikipedia.org/wiki/Dot-com_bubble
Chris Locke: http://rageboy.com
Rick Levine: https://twitter.com/ricklevine
David Weinberger: http://www.hyperorg.com/blogger
The Cluetrain Manifesto: http://cluetrain.com
The entire original text of The Cluetrain Manifesto: http://cluetrain.com/book
“What The Cluetrain Manifesto Teaches Us On Social Media...11 Years Later”:
http://visionarymarketing.com/en/blog/2010/02/what-the-cluetrain-manifesto-teaches-us-
on-social-media-11-years-later
The Cluetrain Legacy and Social Media: http://www.chrisg.com/cluetrain-social-media
Invasion of the Body Snatchers: http://en.wikipedia.org/wiki/Invasion_of_the_Body_Snatchers
CC0 1.0 Universal (CC0 1.0) Public Domain Dedication:
http://creativecommons.org/publicdomain/zero/1.0
e. res on Flickr: https://www.flickr.com/photos/iamtheloop
Creative Commons Attribution 2.0 Generic License: https://creativecommons.org/licenses/by/2.0
Backchannel: New Clues:
https://medium.com/backchannel/internet-under-fire-gets-new-manifests-207a922b459e
Steven Levy: http://www.stevenlevy.com
John Johnston: http://johnjohnston.info
German Translation: http://conceptbakery.de/blog/2015/01/11/new-clues-deutsche-
uebersetzung-die-neuen-thesen-von-den-verfassern-des-cluetrain-manifest
Italian Translation 1:
https://medium.com/bee-free-the-social-bee/cluetrain-15-anni-dopo-9d6b4def4d57
Italian Translation 2: https://medium.com/@nuovetesi/nuove-tesi-4a1def360351
Catalan Translation: https://ca.wikisource.org/wiki/New_clues
“New Clues: Calling on everyone to be Dutiful Individuals” by JP Rangaswami:
http://confusedofcalcutta.com/2015/01/11/new-clues-calling-on-everyone-to-be-dutiful-individuals
Gillmor Gang: Kind of Clue: http://techcrunch.com/2015/01/10/gillmor-gang-kind-of-clue
Facebook Discussion Page: https://www.facebook.com/login.
php?next=https%3A%2F%2Fwww.facebook.com%2Fgroups%2Fnewclues%2F
Disrupt Conference: http://techcrunch.com/event-type/disrupt
Clayton Christensen: http://www.claytonchristensen.com
Disruptive Innovation: http://www.claytonchristensen.com/key-concepts
“Snow on the Water”: http://blogs.law.harvard.edu/doc/2014/08/03/snow-on-the-water
“The World Live Web” by Doc Searls in the October 2005 issue of LJ:
http://www.linuxjournal.com/article/8549

WWW.LINUXJOURNAL.COM / MARCH 2015 / 107

LJ251-March2015.indd 107 2/19/15 9:23 AM


Instant Access to Premium
Online Drupal Training
Instant access to hundreds of hours of Drupal
training with new videos added every week!

Learn from industry experts with real world


H[SHULHQFHEXLOGLQJKLJKSURȴOHVLWHV

Learn on the go wherever you are with apps


for iOS, Android & Roku

We also offer group accounts. Give your


whole team access at a discounted rate!

Learn about our latest video releases and


RIIHUVȴUVWE\IROORZLQJXVRQ)DFHERRNDQG
7ZLWWHU #GUXSDOL]HPH 

Go to http://drupalize.me and
get Drupalized today!

LJ251-March2015.indd 108 2/19/15 9:23 AM