Korn Shell (ksh)

Programming
Tutorial
Philip Brown

This material is copyrighted by Philip Brown, © January 2002-2009

Contents
1. Introduction......................................................................................................................................... 4
1.1. Why scripting?.............................................................................................................................. 4
1.2. What can you do with ksh? .......................................................................................................... 4
1.3. Why ksh, not XYZsh for programming?........................................................................................ 5
2. Ksh preparedness ................................................................................................................................ 5
2.1. Step 0. Learn how to type. ........................................................................................................... 5
2.2. Step 1. Learn how to use a text editor. ........................................................................................ 5
2.3. Step 2. Understand variables. ...................................................................................................... 6
2.4. Step 3. Put everything in appropriate variables........................................................................... 6
2.5. Step 4. Know your quotes ............................................................................................................ 7
3. Ksh basics............................................................................................................................................. 8
3.1. Conditionals.................................................................................................................................. 8
3.1.1. IF ............................................................................................................................................ 8
3.1.2. CASE....................................................................................................................................... 8
3.2. Loops ............................................................................................................................................ 9
3.2.1. WHILE .................................................................................................................................... 9
3.2.2. UNTIL ..................................................................................................................................... 9
3.2.3. FOR ........................................................................................................................................ 9
4. Advanced variable usage................................................................................................................... 10
4.1. Braces ......................................................................................................................................... 10
4.2. Arrays.......................................................................................................................................... 10
4.3. Special variables ......................................................................................................................... 11
4.4. Tweaks with variables ................................................................................................................ 11
5. Ksh and POSIX utilities....................................................................................................................... 12
5.1. cut............................................................................................................................................... 12
5.2. join.............................................................................................................................................. 12
5.3. comm.......................................................................................................................................... 12
5.4. fmt .............................................................................................................................................. 13
5.5. grep and egrep ........................................................................................................................... 13
5.6. sed .............................................................................................................................................. 13
5.7. awk ............................................................................................................................................. 14
6. Ksh Functions..................................................................................................................................... 14
6.1. Why are functions critical?......................................................................................................... 14

1
 6.2. A trivial function ......................................................................................................................... 15
6.3. Debugging your functions .......................................................................................................... 15
6.4. CRITICAL ISSUE: exit vs return .................................................................................................... 15
6.5. CRITICAL ISSUE: "scope" for function variables! ........................................................................ 16
6.6. Write Comments! ....................................................................................................................... 17
7. Ksh built-in functions......................................................................................................................... 17
7.1. Read and Set............................................................................................................................... 18
7.2. The test function ........................................................................................................................ 18
7.3. Built-in math............................................................................................................................... 18
8. Redirection and Pipes........................................................................................................................ 19
8.1. Redirection ................................................................................................................................. 19
8.2. Inline redirection ........................................................................................................................ 20
9. Pipes .................................................................................................................................................. 21
9.1. Combining pipes and redirection ............................................................................................... 21
9.2. Indirect redirection (Inline files)................................................................................................. 21
10. Other Stuff....................................................................................................................................... 22
10.1. eval ........................................................................................................................................... 22
10.2. Backticks ................................................................................................................................... 22
10.3. Text positioning/color games................................................................................................... 23
10.4. Number-based menus.............................................................................................................. 23
10.5. Raw TCP access......................................................................................................................... 24
10.6. Graphics and ksh ...................................................................................................................... 24
11. Paranoia, and good programming practices ................................................................................... 24
11.1. Comment your code................................................................................................................. 25
11.2. INDENT! .................................................................................................................................... 25
11.3. Error checking........................................................................................................................... 26
11.4. cron job paranoia ..................................................................................................................... 27
12. Example of script development....................................................................................................... 28
12.1. The good, the bad, and the ugly............................................................................................... 28
12.2. The newbie progammer version .............................................................................................. 28
12.3. The sysadmin-in-training version ............................................................................................. 29
12.4. The Senior Admin version ........................................................................................................ 29
12.5. The Senior Systems Programmer version ................................................................................ 30
13. Summary of positive features ......................................................................................................... 33

2
3
1. Introduction
This is the top level of my "Intro to Korn shell programming" tree. Korn shell is a 'shell-
scripting' language, as well as a user-level login shell. It is also a superset of a POSIX.1
compliant shell, which is great for ensuring portability.

1.1. Why scripting?

Scripting, when done right, is a fast, easy way to "get the job done", without the usual
"code,compile,test,debug" overhead of writing in C or some other compiled language. It is
easier than C for multiple reasons:

1. Scripting commands tend to be more readable than low-level code. (with the exception
of perl)
2. Scriping languages tend to come with powerful tools attached
3. There is no "compile" phase, so "tweaking" can be done rapidly.

UNIX tends to take #2 to extremes, since it comes standard with "powerful tools" that can be
strung together with pipes or other mechanisms, to get the result you want, with a short
development time. It may not be as efficient as a fully compiled solution, but quite often it can
"get the job done" in a few seconds of run time, compared to 1 second or less for a compiled
program.

A quick scripting solution can be used as a prototype. Then, when you have been using the
prototype happily for a while, and you have evolved the behaviour your end users are happiest
with, you can go back and code a faster, more robust solution in a lower-level language.

That is not to say that scripts cannot be robust! It is possible to do a great deal of error
checking in scripts. Unfortunately, it is not common practice to do so.

1.2. What can you do with ksh?

A heck of a lot!

You have access to the full range of UNIX utilities, plus some nifty built-in resources.

Generally speaking, UNIX scripting is a matter of using the various command line utilities as
appropriate, with the shell as a means of facilitating communication between each step.

Unfortunately, running all these separate 'external' programs can sometimes result in things
working rather slowly. Which is why ksh has a few more things "built in" to it than the older
'sh'.

4
1.3. Why ksh, not XYZsh for programming?
Bourne shell has been THE "standard" UNIX shellscript language for many years. However,
it does lack some things that would be very useful, at a basic level. Some of those things were
added to C-shell (csh) later on. However, csh is undesirable to use in programming, for
various reasons.

Happily, ksh adds most of the things that people say csh has, but sh does not. So much so, that
ksh became the basis for the "POSIX shell". Which means that all properly POSIX-compliant
systems MUST HAVE something compatible, even though it is now named "sh" like the old
Bourne shell. (For example, /usr/xpg4/bin/sh, is a LINK to /bin/ksh, on solaris!) More
precisely, the behaviour of a POSIX-compliant shell is specified in "IEEE POSIX 1003.2"

BTW: "Korn shell" was written by "David Korn", around 1982 at AT&T labs. You can now
freely download the full source from AT&T if you're not lucky enough to use an OS that
comes with ksh already. It's "open source", even!

2. Ksh preparedness
Howdy, and welcome to the intro to Korn shell programming, AKA POSIX.1 shell.

This is the first part of the larger tutorial. It assumes you want to learn how to really be a
programmer in ksh, as opposed to someone who just quickly throws something together in a
file and stops as soon as it works.

This particular chapter assumes you have done minimal or no sh programming before, so has
a lot more general stuff. Here are the most important things to know and do, before really
getting serious about shellscripting.

2.1. Step 0. Learn how to type.

No, I mean type PROPERLY, and unconciously. You should be able to type faster than you
can write, and not be sweating while doing so. The reason for this is that good programming
practices involve extra typing. You won't follow the best practices, if you are too fussed up
about "How can I use as few characters as possible to get all this typing done?"

2.2. Step 1. Learn how to use a text editor.

No, not microsoft word, or word perfect. Those are "word processors". I'm talking about
something small, fast, and quick, that makes shoving pure text around easy. [Note that emacs
does not fit either 'small', 'fast', OR 'quick' :-)]

You will have to be very comfortable with your choice of text editor, because thats how you
make shellscripts. All examples given should be put into some file. You can then run it with
"ksh file".

5
Or, do the more official way; Put the directions below, exactly as-is, into a file, and follow the
directions in it.

#!/bin/ksh
# the above must always be the first line. But generally, lines
# starting with '#' are comments. They dont do anything.
# This is the only time I will put in the '#!/bin/ksh' bit. But
# EVERY EXAMPLE NEEDS IT, unless you want to run the examples with
# 'ksh filename' every time.
#
# If for some odd reason, you dont have ksh in /bin/ksh, change
# the path above, as appropriate.
#
# Then do 'chmod 0755 name-of-this-file'. After that,
# you will be able to use the filename directly like a command

echo Yeup, you got the script to work!

2.3. Step 2. Understand variables.

Hopefully, you already understand the concept of a variable. It is a place you can store a value
to, and then do operations on "whatever is in this place",vs the value directly.

In shellscripts, a variable can contain a collection of letters and/or numbers [aka a 'string'] , as
well as pure numbers.

You set a variable by using

variablename="some string here"

OR
variablename=1234
You access what is IN a variable, by putting a dollar-sign in front of it.
echo $variablename
OR
echo ${variablename}
If you have JUST a number in a variable, you can do math operations on it. But that comes
later on in this tutorial.

2.4. Step 3. Put everything in appropriate variables

Well, okay, not EVERYTHING :-) But properly named variables make the script more easily
readable. There isn't really a 'simple' example for this, since it is only "obvious" in large
scripts. So either just take my word for it, or stop reading and go somewhere else now!

An example of "proper" variable naming practice:

#Okay, this script doesnt do anything useful, it is just for demo purposes.
# and normally, I would put in more safety checks, but this is a quickie.
INPUTFILE="$1"
USERLIST="$2"
OUTPUTFILE="$3"

6
count=0

while read username ; do

grep $username $USERLIST >>$OUTPUTFILE
count=$(($count+1))
done < $INPUTFILE
echo user count is $count

While the script may not be totally readable to you yet, I think you'll agree it is a LOT clearer
than the following;

i=0
while read line ; do
grep $line $2 >> $3
i=$(($i+1))
done <$1
echo $i

Note that '$1' means the first argument to your script.

'$*' means "all the arguments together
'$#' means "how many arguments are there?"

2.5. Step 4. Know your quotes

It is very important to know when, and what type, of quotes to use.
Quotes are generally used to group things together into a single entity.

Single-quotes are literal quotes.

Double-quotes can have their contents expanded

echo "$PWD"
prints out your current directory
echo '$PWD'
prints out the string $PWD
echo $PWDplusthis
prints out NOTHING. There is no such variable "PWDplusthis
echo "$PWD"plusthis
prints out your current directory, and the string "plusthis" immediately following it. You
could also accomplish this with the alternate form of using variables,
echo ${PWD}plusthis

There is also what is sometimes called the `back quote`, or ` backtick`: This is not used to
quote things, but actually to evaluate and run commands.

7
3. Ksh basics
This is a quickie page to run through basic "program flow control" commands, if you are
completely new to shell programming. The basic ways to shape a program, are loops, and
conditionals. Conditionals say "run this command, IF some condition is true". Loops say
"repeat these commands" (usually, until some condition is met, and then you stop repeating).

3.1. Conditionals
3.1.1. IF

The basic type of condition is "if".

if [ $? -eq 0 ] ; then
print we are okay
else
print something failed
fi
IF the variable $? is equal to 0, THEN print out a message. Otherwise (else), print out a
different message. FYI, "$?" checks the exit status of the last command run.

The final 'fi' is required. This is to allow you to group multiple things together. You can have
multiple things between if and else, or between else and fi, or both.
You can even skip the 'else' altogether, if you dont need an alternate case.

if [ $? -eq 0 ] ; then
print we are okay
print We can do as much as we like here
fi

3.1.2. CASE

The case statement functions like 'switch' in some other languages. Given a particular variable,
jump to a particular set of commands, based on the value of that variable.

While the syntax is similar to C on the surface, there are some major differences;

 The variable being checked can be a string, not just a number

 There is no "fall through" with ;;. You hit only one set of commands.. UNLESS you
use ";&" instead of ";;'
 To make up for no 'fall through', you can 'share' variable states
 You can use WILDCARDS to match strings

echo input yes or no

read answer
case $answer in
yes|Yes|y)
echo got a positive answer
# the following ';;' is mandatory for every set
# of comparative xxx) that you do
;;
no)

8
 echo got a 'no'
;;
q*|Q*)
#assume the user wants to quit
exit
;;

*)
echo This is the default clause. we are not sure why or
echo what someone would be typing, but we could take
echo action on it here
;;
esac

3.2. Loops
3.2.1. WHILE

The basic loop is the 'while' loop; "while" something is true, keep looping.

There are two ways to stop the loop. The obvious way is when the 'something' is no longer
true. The other way is with a 'break' command.

keeplooping=1;
while [[ $keeplooping -eq 1 ]] ; do
read quitnow
if [[ "$quitnow" = "yes" ]] ; then
keeplooping=0
fi
if [[ "$quitnow" = "q" ]] ; then
break;
fi
done

3.2.2. UNTIL

The other kind of loop in ksh, is 'until'. The difference between them is that 'while' implies
looping while something remains true.
'until', implies looping until something false, becomes true
until [[ $stopnow -eq 1 ]] ; do
echo just run this once
stopnow=1;
echo we should not be here again.
done

3.2.3. FOR

A "for loop", is a "limited loop". It loops a specific number of times, to match a specific
number of items. Once you start the loop, the number of times you will repeat is fixed.

The basic syntax is

9
for var in one two three ; do
echo $var
done
Whatever name you put in place of 'var', will be updated by each value following "in". So the
above loop will print out
one
two
three
But you can also have variables defining the item list. They will be checked ONLY ONCE,
when you start the loop.
list="one two three"
for var in $list ; do
echo $var
# Note: Changing this does NOT affect the loop items
list="nolist"
done
The two things to note are:

1. It stills prints out "one" "two" "three"

2. Do NOT quote "$list", if you want the 'for' command to use multiple items

If you used "$list" in the 'for' line, it would print out a SINGLE LINE, "one two three"

4. Advanced variable usage

4.1. Braces
Sometimes, you want to immediately follow a variable with a string. This can cause issues if
you use the typical grammar of "echo $a" to use a variable, since ksh by default, attempts to
do "intelligent" parsing of variable names, but it cannot read your mind.
Compare the difference in output in the following lines:
two=2
print one$twothree
print one${two}three
There is no variable named "twothree", so ksh defaults it to an empty value, for the first print
line. However, when you use braces to explicitly show ksh {this is the variable name}, it
understands that you want the variable named "two", to be expanded in the middle of those
other letters.

4.2. Arrays
Yes, you CAN have arrays in ksh, unlike old bourne shell. The syntax is as follows:
# This is an OPTIONAL way to quickly null out prior values
set -A array
#
array[1]="one"
array[2]="two"
array[3]="three"
three=3

print ${array[1]}

10
print ${array[2]}
print ${array[3]}
print ${array[three]}

4.3. Special variables

There are some "special" variables that ksh itself gives values to. Here are the ones I find
interesting

 PWD - always the current directory

 RANDOM - a different number every time you access it
 $$ - the current process id (of the script, not the user's shell)
 PPID - the "parent process"s ID. (BUT NOT ALWAYS, FOR FUNCTIONS)
 $? - exit status of last command run by the script
 PS1 - your "prompt". "PS1='$PWD:> '" is interesting.
 $1 to $9 - arguments 1 to 9 passed to your script or function (you can actually have
higher, but you need to use braces for those)

4.4. Tweaks with variables

Both bourne shell and KSH have lots of strange little tweaks you can do with the ${} operator.
The ones I like are below.

To give a default value if and ONLY if a variable is not already set, use this construct:

APP_DIR=${APP_DIR:-/usr/local/bin}

(KSH only)
You can also get funky, by running an actual command to generate the value. For example

DATESTRING=${DATESTRING:-$(date)}

(KSH only)
To count the number of characters contained in a variable string, use ${#varname}.

echo num of chars in stringvar is ${#stringvar}

11
5. Ksh and POSIX utilities
POSIX.1 (or is it POSIX.2?) compliant systems (eg: most current versions of UNIX) come
with certain incredibly useful utilities. The short list is:
cut, join, comm, fmt, grep, egrep, sed, awk

Any of these commands (and many others) can be used within your shellscripts to manupulate
data.

Some of these are programming languages themselves. Sed is fairly complex, and AWK is
actually its own mini-programming language. So I'll just skim over some basic hints and
tricks.

5.1. cut
"cut" is a small, lean version of what most people use awk for. It will "cut" a file up into
columns, and particularly, only the columns you specify. Its drawbacks are:

1. It is picky about argument order. You MUST use the -d argument before the -f
argument
2. It defaults to a tab, SPECIFICALLY, as its delimiter of columns.

The first one is just irritating. The second one is a major drawback, if you want to be flexible
about files. This is the reason why AWK is used more, even for this trivial type of operator:
Awk defaults to letting ANY whitespace define columns.

5.2. join
join is similar to a "database join" command, except it works with files. If you have two files,
both with information sorted by username, you can "join" them in one file, IF and ONLY IF
they are also sorted by that same field. For example
john_s John Smith
in one file, and
john_s 1234 marlebone rd
will be joined to make a single line,
john_s John Smith 1234 marlebone rd

If the files do not already have a common field, you could either use the paste utility to join
the two files, or give each file line numbers before joining them, with

cat -n file1 >file1.numbered

5.3. comm
I think of "comm" as being short for "compare", in a way. But technically, it stands for
"common lines". First, run any two files through "sort". Then you can run 'comm file1 file2' to
tell you which lines are ONLY in file1, or ONLY in file2, or both. Or any combination.

12
For example

comm -1 file1 file2

means "Do not show me lines ONLY in file1." Which is the same thing as saying "Show me
lines that are ONLY in file2", and also "Show me lines that are in BOTH file1 and file2".

5.4. fmt
fmt is a simple command that takes some informational text file, and word-wraps it nicely to
fit within the confines of a fixed-width terminal. Okay, it isn't so useful in shellscripts, but its
cool enough I just wanted to mention it :-)

pr is similarly useful. But where fmt was more oriented towards paragaphs, pr is more
specifically toward page-by-page formatting.

5.5. grep and egrep

These are two commands that have more depth to them than is presented here. But generally
speaking, you can use them to find specific lines in a file (or files) that have information you
care about.
One of the more obvious uses of them is to find out user information. For example,
grep joeuser /etc/passwd
will give you the line in the passwd file that is about account 'joeuser'. If you are suitable
paranoid, you would actually use
grep '^joeuser:' /etc/passwd
to make sure it did not accidentally pick up information about 'joeuser2' as well.

(Note: this is just an example: often, awk is more suitable than grep, for /etc/passwd fiddling)

5.6. sed
Sed actually has multiple uses, but its simplest use is "substitute this string, where you see that
string". The syntax for this is
sed 's/oldstring/newstring/'

This will look at every line of input, and change the FIRST instance of "oldstring" to
"newstring".

If you want it to change EVERY instance on a line, you must use the 'global' modifier at the
end:

sed 's/oldstring/newstring/g'

If you want to substitute either an oldstring or a newstring that has slashes in it, you can use a
different separator character:

sed 's:/old/path:/new/path:'

13
5.7. awk
Awk really deserves its own tutorial, since it is its own mini-language. And, it has one!

But if you dont have time to look through it, the most common use for AWK is to print out
specific columns of a file. You can specify what character separates columns. The default is
'whitespace' (space, or TAB). But the cannonical example, is "How do I print out the first and
fifth columns/fields of the password file?"

awk -F: '{print $1,$5}' /etc/passwd

"-F:" defines the "field separator" to be ':'

The bit between single-quotes is a mini-program that awk interprets. You can tell awk
filename(s), after you tell it what program to run. OR you can use it in a pipe.

You must use single-quotes for the mini-program, to avoid $1 being expanded by the shell
itself. In this case, you want awk to literally see '$1'

"$x" means the 'x'th column

The comma is a quick way to say "put a space here".
If you instead did

awk -F: '{print $1 $5}' /etc/passwd

awk would not put any space between the columns!

If you are interested in learning more about AWK, read my AWK tutorial

6. Ksh Functions
Functions are the key to writing just about ANY program that is longer than a page or so of
text. Other languages may call functions something else. But essentially, its all a matter of
breaking up a large program, into smaller, managable chunks. Ideally, functions are sort of
like 'objects' for program flow. You pick a part of your program that is pretty much self-
contained, and make it into its own 'function'

6.1. Why are functions critical?

Properly written functions can exist by themselves, and affect only small things external to
themselves. You should DOCUMENT what things it changes external to itself. Then you can
look very carefully just at the function, and determine whether it actually does what you think
it does :-)

14
When your program isn't working properly(WHEN, not if), you can then put in little debug
notes to yourself in the approximate section you think is broken. If you suspect a function is
not working, then all you have to verify is

 Is the INPUT to the function correct?

 Is the OUTPUT from the function correct?

Once you have done that, you then know the entire function is correct, for that particular set
of input(s), and you can look for errors elsewhere.

6.2. A trivial function

printmessage() {
echo "Hello, this is the printmessage function"
}

printmessage

The first part, from the first "printmessage()" all the way through the final '}', is the function
definition. It only defines what the function does, when you decide to call it. It does not DO
anything, until you actually say "I want to call this function now".

You call a function in ksh, by pretending it is a regular command, as shown above. Just have
the function name as the first part of your line. Or any other place commands go. For example,

echo The message is: `printmessage`

Remember: Just like its own separate shellscript. Which means if you access "$1" in a
function, it is the first argument passed in to the function, not the shellscript.

6.3. Debugging your functions

If you are really really having difficulties, it should be easy to copy the entire function into
another file, and test it separately from the main program.

This same type of modularity can be achived by making separate script files, instead of
functions. In some ways, that is almost preferable, because it is then easier to test each part by
itself. But functions run much faster than separate shellscripts.

A nice way to start a large project is to start with multiple, separate shellscripts, but then
encapsulate them into functions in your main script, once you are happy with how they work.

6.4. CRITICAL ISSUE: exit vs return

THE main difference when changing between shellscripts and functions, is the use of "exit".

15
'exit' will exit the entire script, whether it is in a function or not.
'return' will just quit the function. Like 'exit', however, it can return the default "sucess" value
of 0, or any number from 1-255 that you specify. You can then check the return value of a
function, just in the same way you can check the return value of an external program, with the
$? variable.

# This is just a dummy script. It does not DO anything

fatal(){
echo FATAL ERROR
# This will quit the 'fatal' function, and the entire script that
# it is in!
exit
}

lessthanfour(){
if [[ "$1" = "" ]] ; then echo "hey, give me an argument" ; return
1; fi

# we should use 'else' here, but this is just a demonstration

if [[ $1 -lt 4 ]] ; then
echo Argument is less than 4
# We are DONE with this function. Dont do anything else in
# here. But the shellscript will continue at the caller
return
fi

echo Argument is equal to or GREATER than 4

echo We could do other stuff if we wanted to now
}

echo note that the above functions are not even called. They are just
echo defined

A bare "return" in a shellscript is an error. It can only be used inside a function.

6.5. CRITICAL ISSUE: "scope" for function variables!

Be warned: Functions act almost just like external scripts... except that by default, all
variables are SHARED between the same ksh process! If you change a variable name inside a
function.... that variable's value will still be changed after you have left the function!! Run this
script to see what I mean.
#!/bin/sh
# Acts the same with /bin/sh, or /bin/ksh, or /bin/bash
subfunc(){
echo sub: var starts as $var
var=2
echo sub: var is now $var
}
var=1
echo var starts as $var, before calling function '"subfunc"'
subfunc # calls the function
echo var after function is now $var
To avoid this behaviour, and give what is known as "local scope" to a variable, you can use
the typeset command, to define a variable as local to the function.
#!/bin/ksh

16
# You must use a modern sh like /bin/ksh, or /bin/bash for this
subfunc(){
typeset var
echo sub: var starts as $var '(empty)'
var=2
echo sub: var is now $var
}
var=1
echo var starts as $var, before calling function '"subfunc"'
subfunc # calls the function
echo var after function is now $var
Another exception to this is if you call a function in the 'background', or as part of a pipe (like
echo val | function )
This makes the function be called in a separate ksh process, which cannot dynamically share
variables back to the parent shell. Another way that this happens, is if you use backticks to
call the function. This treats the function like an external call, and forks a new shell. This
means the variable from the parent will not be updated. Eg:
func() {
newval=$(($1 + 1))
echo $newval
echo in func: newval ends as $newval
}
newval=1
echo newval in main is $newval
output=`func $newval`
func $newval
echo output is : $output
echo newval finishes in main as $newval

6.6. Write Comments!

Lastly, as mentioned in the good practices chapter, dont forget to comment your functions!
While shellscripts are generally easier to read than most programming languages, you really
can't beat actual human language to explain what a function is doing!

7. Ksh built-in functions

Calling the many UNIX programs in /bin and elsewhere can be very useful. The one
drawback is speed. Every time you call a separate program, there is a lot of overhead in
starting it up. So the conciencious programmer always tries to use built-in functions over
external ones. In particular, ksh programmers should always try use '[[ ]]' over '[ ]', except
where [] is neccessary

The more useful functions in ksh I find are:

 'read' and 'set' functions

 built-in 'test': [[ ]]
(But this is apparently NOT a part of POSIX!!)
 built-in math : $(( ))
 built-in 'typeset'

17
See the manpages for 'test' and 'typeset', if you want full info on those beasties.

7.1. Read and Set

read varname
will set the variable varname to have the value of the next line read in from standard input.

What often comes next, is

set $varname
This sets the argument variables $1, $2, etc to be set as if the program were called with
$varname as the argument string to the shellscript. So, if the value of varname is "first second
third", then $1="first", $2="second", and $3="third".

Note that if you want to access "double-digit" arguments, you cannot use "$10". it will get
interpreted as "$1,""0". To access argument #10 and higher you must explicitly define the
limits of the variable string, with braces:

echo ${10}
This is also good to know, if you wish to follow a variable immediately followed by a string.
Compare the output from the following lines:
a="A "
echo $astring
echo ${a}string

7.2. The test function

In brief, 'test' can let you check the status of files OR string values.
Here are the most common uses for it.
if [[ $? -ne 0 ]] ; then echo status is bad ; fi
if [[ "$var" != "good" ]] ; then echo var is not good ; fi
if [[ ! -f /file/name ]] ; then echo /file/name is not there ; fi
if [[ ! -d /dir/name ]] ; then echo /dir/name is not a directory ; fi

Please note that [[]] is a special built-in version of test, that is almost, but not 100%, like the
standard []. The main difference being that wildcard expansion does not work within [[]].

7.3. Built-in math

The math evaluator is very useful. Everything inside the double-parens gets evaluated with
basic math functions. For example;

four=$((2 + 2))
eight=$(($four + 4))
print $(($four * $eight))

Warning: Some versions of ksh allow you to use floating point with $(()). Most do NOT.

Also, be wary of assumptions. Being "built in" is not always faster than an external progam.
For example, it is trivial to write a shell-only equivalent of the trivial awk usage, "awk '{print
$2}'", to print the second column. However, compare them on a long file:
18
# function to emulate awk '{print $2}'
sh_awk(){
while read one two three ; do
print $two
done
}

# and now, compare the speed of the two methods

time sh_awk </usr/dict/words >/dev/null

time awk '{print $2}' </usr/dict/words >/dev/null

The awk version will be much much faster. This is because ksh scripts are interpreted, each
and every time it executes a line. AWK, however, loads up its programming in one go, and
figures out what it is doing ONE TIME. Once that overhead has been put aside, it then can
repeat its instructions very fast.

8. Redirection and Pipes

There are lots of strange and interesting ways to connect utilities together. Most of these you
have probably already seen.

The standard redirect to file;

ls > /tmp/listing

and piping output from one command to another

ls | wc -l

But bourne-shell derivatives give you even more power than that.

Most properly written programs output in one of two ways.

1. Progress messages go to stdout, error messages go to stderr

2. Data goes to stdout, error AND progress messsages go to stderr

If you know which of the categories your utilities fall into, you can do interesting things.

8.1. Redirection
An uncommon program to use for this example is the "fuser" program under solaris. it gives
you a long listing of what processes are using a particular file. For example:

$ fuser /bin/sh
/bin/sh: 13067tm 21262tm

19
If you wanted to see just the processes using that file, you might initially groan and wonder
how best to parse it with awk or something. However, fuser actually splits up the data for you
already. It puts the stuff you may not care about on stderr, and the meaty 'data' on stdout. So if
you throw away stderr, with the '2>' special redirect, you get
$ fuser /bin/sh 2>/dev/null
13067 21262

which is then trivially usable.

Unfortunately, not all programs are that straightforward :-) However, it is good to be aware of
these things, and also of status returns. The 'grep' command actually returns a status based on
whether it found a line. The status of the last command is stored in the '$?' variable. So if all
you care about is, "is 'biggles' in /etc/hosts?" you can do the following:

grep biggles /etc/hosts >/dev/null

if [[ $? -eq 0 ]] ; then
echo YES
else
echo NO
fi
As usual, there are lots of other ways to accomplish this task, even using the same 'grep'
command. However, this method has the advantage that it does not waste OS cycles with a
temp file, nor does it waste memory with a potentially very long variable.
(If you were looking for something that could potentially match hundreds of lines, then
var=`grep something /file/name` could get very long)

8.2. Inline redirection

You have seen redirection TO a file. But you can also redirect input, from a file. For programs
that can take data in stdin, this is useful. The 'wc' can take a filename as an argument, or use
stdin. So all the following are roughly equivalent in result, although internally, different
things happen:
wc -l /etc/hosts
wc -l < /etc/hosts
cat /etc/hosts | wc -l

Additionally, if there are a some fixed lines you want to use, and you do not want to bother
making a temporary file, you can pretend part of your script is a separate file!. This is done
with the special '<<' redirect operator.

command << EOF

means, "run 'command', but make its stdin come from this file right here, until you see the
string 'EOF'"

EOF is the traditional string. But you can actually use any unique string you want.
Additionally, you can use variable expansion in this section!

DATE=`date`
HOST=`uname -n`
mailx -s 'long warning' root << EOF

20
Something went horribly wrong with system $HOST
at $DATE
EOF

9. Pipes
In case you missed it before, pipes take the output of one command, and put it on the input of
another command. You can actually string these together, as seen here;
grep hostspec /etc/hosts| awk '{print $1}' | fgrep '^10.1.' | wc -l

This is a fairly easy way to find what entries in /etc/hosts both match a particular pattern in
their name, AND have a particular IP address ranage.

The "disadvantage" to this, is that it is very wasteful. Whenever you use more than one pipe at
a time, you should wonder if there is a better way to do it. And indeed for this case, there most
certainly IS a better way:

grep '^10\.1\..*hostspec' /etc/hosts | wc -l

There is actually a way to do this with a single awk command. But this is not a lesson on how
to use AWK!

9.1. Combining pipes and redirection

An interesting example of pipes with stdin/err and redirection is the "tar" command. If you
use "tar cvf file.tar dirname", it will create a tar file, and print out all the names of the files in
dirname it is putting in the tarfile. It is also possible to take the same 'tar' data, and dump it to
stdout. This is useful if you want to compress at the same time you are archiving:
tar cf - dirname | compress > file.tar.Z
But it is important to note that pipes by default only take the data on stdout! So it is possible
to get an interactive view of the process, by using
tar cvf - dirname | compress > file.tar.Z
stdout has been redirected to the pipe, but stderr is still being displayed to your terminal, so
you will get a file-by-file progress report. Or of course, you could redirect it somewhere else,
with
tar cvf - dirname 2>/tmp/tarfile.list | compress > file.tar.Z

9.2. Indirect redirection (Inline files)

Additionally, there is a special type of pipes+redirection. This only works on systems with
/dev/fd/X support. You can automatically generate a "fake" file as the result of a command
that does not normally generate a file. The name of the fake files will be
/dev/fd/{somenumberhere}

Here's an example that doesnt do anything useful

wc -l <(echo one line) <(echo another line)

21
wc will report that it saw two files, "/dev/fd/4", and "/dev/fd/5", and each "file" had 1 line
each. From its own perspective, wc was called simply as
wc -l /dev/fd/4 /dev/fd/5

There are two useful components to this:

1. You can handle MULTIPLE commands' output at once

2. It's a quick-n-dirty way to create a pipeline out of a command that "requires" a
filename (as long as it only processes its input in a single continuous stream).

10. Other Stuff

And here's stuff that I cant fit anywhere else :-)

 eval
 Backticks
 Text positioning/color/curses stuff
 Number-based menus
 Raw TCP access
 Graphics and ksh

10.1. eval
The eval command is a way to pretend you type something directly. This is a very dangerous
command. Think carefully before using it.

One way of using eval, is to use an external command to set variables that you do not know
the name of beforehand. Or a GROUP of variables. A common use of this, is to set terminal-
size variables on login:

eval `resize`

10.2. Backticks
There are ways to put the output of one command as the command line of another one. There
are two methods of doing this that are basically equivalent:
echo This is the uptime: `uptime`
echo This is the uptime: $(uptime)
Technically, the second one is the POSIX-preferred one.

In addition to creating dynamic output, this is also very useful for setting variables:

datestring=`date`
echo "The current date is: $datestring"

22
10.3. Text positioning/color games
This is actually a huge topic, and almost deserves its own tutorial. But I'm just going to
mention it briefly.

Some people may be familiar with the "curses" library. It is a way to manipulate and move
around text on a screen, reguardless of what kind of "terminal" the user is using.

As mentioned, this is a potentially huge topic. So, I'm just going to give you a trivial example,
and say "Go read the man-page on tput". Well, okay, actually, you have to read the "tput"
manpage, AND either the "terminfo" or "termcap" manpage to figure out what magical 3-5
letter name to use. For example, it should tell you that "cup" is short for the "cursor_address"
command. But you must use "cup", NOT "cursor_address", with tput.

tput init
tput clear
tput cup 3 2
print -n "Here is a clean screen, with these words near the top"
endline=`tput cols`
tput cup $(($endline - 2))
print "and now, back to you"
sleep 2

The above example clear the screen, prints the given line at a SPECIFIC place on the screen,
then puts the cursor back down near the bottom of the screen for you.

PS: If you've been doing a lot of funky things with the screen, you might want to do a

tput reset
as the last thing before your shellscript exits.

10.4. Number-based menus

You dont have to build your own "choose a number" function: ksh has one already! But note
that it returns the value of the line, not the number of the line.
select word in one two three exit; do
echo word is $word
echo reply is $REPLY
if [[ "$word" = "exit" ]] ; then
break;
fi
done

This will print out a mini-menu like the following:

1) one
2) two
3) three
4) exit
#?

23
Note that this will loop between "do ... done" until you trigger a break somehow! (or until the
user control-c's or whatever). So dont forget an exit condition!

10.5. Raw TCP access

Ksh88 has a built in virtual filesystem that looks like it is under /dev/tcp. You can use it to
create connections to specific ports, if you know the IP address.

Here is a trivial example that just opens up a connection to an SMTP server. Note that the
connection is half-duplex: You do NOT see data that you send to the other side.

#!/bin/ksh -p

MAILHOST=127.0.0.1
exec 3<>/dev/tcp/$MAILHOST/25 || exit 1

read -r BANNER <&3

echo BANNER is $BANNER
print -u3 HELO myhost.com
read -r REPLY <&3
echo REPLY is $REPLY

The output will look something like the following:

BANNER is 220 yourhost.domain.com ESMTP Sendmail 8.11.6+Sun/8.11.6; Tue, 3

Dec 2002 17:30:01 -0800 (PST)
REPLY is 250 yourhost.domain.com Hello localhost [127.0.0.1], pleased to
meet you

Note that we use the "-r" flag to read. In this particular example, it is not neccessary. But in
the general case, it will give you the data "raw". Be warned that if the shell cannot open the
port, it will kill your entire script, with status 1, automatically

You can also dump the rest of the data waiting on the socket, to whereever you like, by doing

cat <&3 >somefile

10.6. Graphics and ksh

Not many people are aware of this, but there is actually a graphical version of ksh, called
"dtksh". It was created as part of "CDE". Any of the modern UNIX(tm)es should come with it,
in /usr/dt/bin/dtksh. If you are interested, take a look at some dtksh demos that someone else
has written. And/or, you might see if you have a /usr/dt/share/examples/dtksh/ directory
present on your machine.

11. Paranoia, and good programming practices

First, a bit of good programming practice:

24
11.1. Comment your code.
You really should at MINIMUM have some comment about every page (that's every 24 lines).
Ideally, you should always comment all your functions. One-line functions can probably stand
by themselves, but otherwise, a quick single line comment is a good thing to have for small
functions.

# This function cleans up the file before printing

pcleanup(){
....
}

For longer functions, you should really use formal comment spec. Something like

# Function xyz
# Usage: xyz arg1 arg2 arg3 arg4
# arg1 is the device
# arg2 is a file
# arg3 is how badly you want to mangle it
# arg4 is an optional output file
# Result: the file will be transferred to the device, with the
# appropriate ioctls. Any err messages will be saved in the output
# file, or to stderr otherwise
xyz(){
...
}
Note that shellscripts are themselves one large "function". So dont forget basic comments on
your shellscript's functionality at the top of the file!

11.2. INDENT!
Every time you start a function, indent.

Every time you are in a while loop, or if clause, indent.

This makes it easier to see at a glance what level you are at.

# top level
print this is the top

somefunction(){
# indent in functions
print we are in the function now

if [[ "$1" != "" ] ; then

# yes, ANOTHER indent!
print "Hey, we have an argument: $1"
print full args are $*
else
# indent again!!
print "oh well. no arguments to play with"
fi

25
 print leaving somefunction now
}

# And now we can clearly see that all this stuff is outside any function.
# This makes it easier to find the "main line" of the script
print original shellscript args are $0
print lets try somefunction
somefunction heres some args

11.3. Error checking

If you are using scripts to check on important systems, or perform crucial functions, it is very
important to provide feedback to the calling process when and if something goes wrong.

The simplest method of doing this, is a simple

exit 1
# (but it would be nice to print out SOME message before exiting!)
Nice programs will notice that your script exited with a non-zero status. [Remember, the
status of the last command is in '$?']
Ideally, they will complain.
On the other hand, sometimes your own scripts are the ones that are doing the calling!

In that type of situation, it may be suitable to have a top-level script that keeps an eye on
things. A simple example is:

fatal(){
# Something went horribly wrong.
# print out an errormessage if provided, then exit with an
# "error" status

if [[ "$1" != "" ]] ; then

print Something went horribly wrong with "$1"
else
print "something went horribly wrong (Somewhere?)"
fi

# You mail also want to SEND EMAIL, or trigger a PAGER or

# something here:
# mailx -s "Arrrg! we failed checks!" admin@yourhost.com <
/dev/null
#
exit 1
}

check_on_modems
if [[ $? -ne 0 ]] ; then fatal modems ; fi

check_on_network
if [[ $? -ne 0 ]] ; then fatal network ; fi

check_on_servers
if [[ $? -ne 0 ]] ; then fatal servers ; fi

26
Note that even my paranoid 'fatal' function, IS PARANOID!
Normally, it is assumed you will call it with "fatal what-failed". But if you somehow dont, it
notices, and provides a default.

Sometimes, making the assumption that $1 contains valid data can completely screw up the
rest of your function or script. So if it is an important function, assume nothing!

This is particularly true of CGI scripts. [Gasp]. Yes, Virginia, it IS possible to write CGI in
something other than perl.

11.4. cron job paranoia

If you are coding a script that specifically is in a cronjob, there are two things to be aware of.
The number one most important thing is;

Set your PATH variable!!!

People always forget this one. It doesnt matter what is in your .profile, cron will reset the
PATH variable to something really short and stupid, like just /usr/bin. So set your
PATH=/whatever:/usr/bin explicitly in your cron scripts.

The second tip is more an advisory note.

cron by default saves anything that gets send to 'stderr', and MAILS IT to the owner of the
cron job. So, sometimes, if you just want a minor error logged somewhere, it is sufficient to
just do

print "Hey, this is kinda wierd" >/dev/fd/2

which will send the output of the print to stderr, which will then go to email. Another way of
doing it, if your system does not have /dev/fd, could be
print "Wow, this is tortured" 1>&2

Contrariwise, if you want to throw away ALL output, use

command >/dev/null 2>&1

If you do not regularly read email for the user in question, you can either set up an alias for
that user, to forward all its email to you, or do

export MAILTO=my@address.here

The MAILTO trick does not work on all cron demons, however.

27
12. Example of script development
12.1. The good, the bad, and the ugly
Hopefully, you have read through all the other chapters by this point. This page will show you
the "big picture" of shellscript writing.

Here are four versions of essentially the same program: a wrapper to edit a file under SCCS
version control.
The basic task is to use the sccs command to "check out" a file under version control, and then
automatically edit the file. The script will then be used by "users", aka coders, who may not
be particularly advanced UNIX users. Hence, the need for a wrapper script.

While the basic functionality is the same across all versions , the differences in safety and
usability between the first version and the last version are staggering.

The first one is extremely bad: it would be written by someone who has just picked up a book
on shellscripting, and has decided, "I'm a programmer now".

The second one is an improvement, showing some consideration to potential users by having
safety checks.

The third one is a good, solid version. It's a positive role model for your scripts.

The final one is a full-blown, paranoid, commented program unto itself, with whiz-bang
features. This is the way a professional programmer would write it. Don't let that put you off:
it's the way you can and should write it too! You might start with a version like the initial
dumb one, as the initial step in your code development, just to make sure you have the basic
functionality for the task. But after it is shown to work, you should upgrade it to a more
reasonable one immediately.

Note: there is a summary of good practices show in this page, at the bottom.

12.2. The newbie progammer version

#!/bin/ksh

sccs edit $1
if [ "$EDITOR" = "" ] ; then
EDITOR=vi
fi
$EDITOR $1

This version makes somewhat of an attempt to be user friendly, by having a check for a user-
specified EDITOR setting, and using it if available. However, there are no comments, no error
checking, and no help for the user whatsoever!

28
12.3. The sysadmin-in-training version
#!/bin/ksh

# Author: Joe Newguy

if [ $# -lt 1 ] ; then
print "This program will check out a file, or files,
with sccs"
exit 1
fi

# set EDITOR var to "vi" if not already set

EDITOR=${EDITOR:-vi}

sccs edit $@
$EDITOR $@

This is somewhat of a step above the prior version. It accepts multiple files as potential
arguments. It's always nice to be flexible about the number of files your scripts can handle. It
also has a usage message, if the script is called without arguments. Plus, it's always a good
idea to put your name in in, unless you're working for the company of "Me, Myself and I,
Inc."

Unfortunately, there is still quite a bit lacking, as you can tell by comparing it to the next
version.

12.4. The Senior Admin version

#!/bin/ksh

# SCCS editing wrapper, version 0.3

# Author - Sys Admin
# Usage: see usage() function, below

usage(){
print sedit - a wrapper to edit files under SCCS
print "usage: sedit file {file2 ...}"
}

# Set EDITOR var to "vi" if not already set to something

EDITOR=${EDITOR:-vi}
# Could already be in path, but it doesnt hurt to add it
again.
# Sorry, I assume solaris machines everywhere: adjust as
needed,
# if your sccs lives somewheres else
SCCSBIN=/usr/ccs/bin
if [ ! -x $SCCSBIN/sccs ] ; then
print ERROR: sccs not installed on this machine.
Cannot continue.
usage
exit 1
fi

29
 PATH=$SCCSBIN:$PATH

if [ $# -lt 1 ] ; then
usage
print ERROR: no files specified
exit 1
fi

# Yes, I could use "sccs edit $@" and check for a single
error, but this
# approach allows for finer error reporting
for f in $@ ; do
sccs edit $f
if [ $? -ne 0 ] ; then
print ERROR checking out file $f
if [ "$filelist" != "" ] ; then
print "Have checked out $filelist"
fi
exit 1
fi
filelist="$filelist $f"
done

$EDITOR $filelist
if [ $? -eq 0 ] ; then
print ERROR: $EDITOR returned error status
exit 1
fi

This guy has been around the block a few times. He's a responsible sysadmin, who likes to be
disaster-prepared. In this case, the most likely "disaster" is 100 calls from developers asking
"Why doesnt it work for me?" So when things break, it's a good idea to provide as much
information as possible to the user.

Nice things to note:

 Sets special variables at the top of the script, in a unified place

 Paranoid checks about EVERYTHING
 Returns an error status from the script, on non-clean exit condition ("exit 1", vs "exit")
 Use of comments. Not only does he specify what he is doing, he clarifies what he is
NOT doing, and why.

Compare and contrast the first version of the program, to this one. Then try to make your own
scripts be more like this!

12.5. The Senior Systems Programmer version

#!/bin/ksh

# SCCS editing wrapper, version 1.3

# Author - Phil Brown

30
# Usage: see usage() function, below

usage(){
print sedit - a wrapper to edit files under SCCS
print "Usage: sedit [-c|-C] [-f] file {file2 ...}"
print " -c check in file(s) after edit is
complete"
print " -C check in all files with single
revision message"
print " -f ignore errors in checkout"
}

# Set EDITOR var to "vi" if not already set to something

EDITOR=${EDITOR:-vi}
# Could already be in path, but it doesnt hurt to add it
again.
# Sorry, I assume solaris machines everywhere: adjust as
needed.
PATH=$PATH:/usr/ccs/bin
if [ ! -x /usr/ccs/bin/sccs ] ; then
print ERROR: sccs not installed on this machine.
Cannot continue.
usage
exit 1
fi

while getopts "cCfh" arg

do
case $arg in
c)
checkin="yes"
;;
C)
checkinall="yes"
;;
f)
force="yes"
;;
h|*)
usage
exit 1
;;
esac
done

shift $(($OPTIND - 1))

if [ $# -lt 1 ] ; then
usage
print ERROR: no files specified
exit 1
fi

if [ "$checkinall" != "" ] && [ "$checkin" != "" ] ; then

print WARNING: -c and -C used. Will use -C.
fi

# Yes, I could use "sccs edit $@" and check for a single
error, but this
# approach allows for finer error reporting.

31
 # "$@" is a special construct that catches spaces in
filenames.
# Note that "$*" is NOT 100% the same thing.
for f in "$@" ; do
sccs edit "$f"
if [ $? -ne 0 ] ; then
print ERROR checking out file $f
if [ "$force" = "" ] ; then
if [ "$filelist" != "" ] ; then
print "Have checked out
$filelist"
fi
exit 1
fi
# else, -f is in effect. Keep going
fi
filelist="$filelist $f"
done

# I would like to use "$filelist", but that does not

preserve spaces
# in file names
$EDITOR "$@"
if [ $? -eq 0 ] ; then
print ERROR: $EDITOR returned error status
exit 1
fi

if [ "$checkinall" != "" ] ; then

# -C option used. re-check in all files at once.
sccs delget $filelist
if [ $? -ne 0 ] ; then
print "ERROR checking in files?"
exit 1
fi
exit 0
fi
if [ "$checkin" != "" ] ; then
# -c option used. re-check in each file.
for file in $filelist ; do
sccs delget $file
if [ $? -ne 0 ] ; then
print "WARNING: failed to check in
$file"
fi
# do NOT stop after error. Keep trying to
check
# in any other files
done
fi

This guy has been around the block a few times. Heck, he helped BUILD the block ;-)

This was originally my third and final version. It's the way I would really write the script. But
I decided it might be a bit daunting to new scripting folks, so I made a new intermediate third
step, above.

Additional things beyond the previous version:

32
  Provides optional extra functionality, where it makes sense. (added -c, -C, and -f
option flags.) This shows understanding of writing scripts, AND understanding of the
area of the task (SCCS version control)
 Use of the 'getopts' standard util, rather than hand-coding a custom argument parser

13. Summary of positive features

Here is a summary of all the positives added through the different versions.

 COMMENT YOUR SCRIPTS!!!

 Check for user-environment variables, where appropriate
 When setting/overriding special variables, do it at the top of the script, in a unified
place
 Accept one OR MORE files as arguments, when it makes sense to do so.
 Have a "usage" message
 Be Paranoid: check return statuses
 Return correct statuses yourself: ("exit 1", vs "exit", on error)
 Take advantage of 'getopts' when adding optional feature flags

33