Thoughtful Solutions Ltd

http://very.thoughtful-solutions.info/

Solaris
Fundamentals
Training

Abstract
Operations Group Solaris Fundamentals Training

Attribution-ShareAlike 2.0 United Kingdom

You are free:

to copy, distribute, display, and perform the work

to make derivative works
to make commercial use of the work
Under the following conditions:

Attribution. You must give the original author credit.

Share Alike. If you alter, transform, or build upon this work, you
may distribute the resulting work only under a licence identical to
this one.
For any reuse or distribution, you must make clear to others the licence
terms of this work.
Any of these conditions can be waived if you get permission from the
copyright holder.

Your fair use and other rights are in no way affected by the above.
This is a human-readable summary of the Legal Code (the full licence).
Disclaimer

Unix History and Lore................................................................................................9

Introduction..........................................................................................................10
What is Unix?.......................................................................................................11
Distributed Computing with Unix........................................................................12
Aims.....................................................................................................................13
Why Unix?...........................................................................................................14
Unix Philosophy...................................................................................................15
Major Features......................................................................................................16
Unix Lore.............................................................................................................17
Potted History.......................................................................................................18
Unix Flavours.......................................................................................................20
Key Concepts............................................................................................................26
The Business of Computing.................................................................................27
Files & Processes..................................................................................................28
Organisation of Files............................................................................................29
Organisation of Processes.....................................................................................30
Login....................................................................................................................31
Login File & Process............................................................................................32
What are Users?....................................................................................................33
Unix Structure......................................................................................................34
Kernel Responsibilities.........................................................................................35
Getting Started: Login..............................................................................................36
CDE Login Screen...............................................................................................37
Command Line Login Screen...............................................................................39
Usernames............................................................................................................40
Valid Usernames..................................................................................................41
Good Passwords...................................................................................................42
Passwords.............................................................................................................43
Changing Passwords [passwd].............................................................................44
Login Failures.......................................................................................................45
Why login fails.....................................................................................................46
Interruptions.........................................................................................................47
Ending the sessions [logout].................................................................................48
/etc/passwd : home of identity..............................................................................49
/etc/shadow : Password vault................................................................................50
/etc/group: where groups live...............................................................................51
Summary..............................................................................................................52
File Hierarchy...........................................................................................................53
The Unix File system............................................................................................54
Hierarchical Structure...........................................................................................55
Directory Paths.....................................................................................................56
File names.............................................................................................................57
What is a file?.......................................................................................................58
Varieties of Files ..................................................................................................59
Traversing the File System : cd............................................................................60
Examining Directory Contents : ls.......................................................................61
Shell Wildcards....................................................................................................62
Building the File System : mkdir/rmdir................................................................63
Copying Files : cp.................................................................................................64
Copying with Wildcards.......................................................................................65
Moving Files : mv................................................................................................66
Deleting Files : rm................................................................................................67

Linking Files.........................................................................................................68
Linking Files : ln..................................................................................................69
Hard links : ln.......................................................................................................70
Symbolic links : ln................................................................................................71
Why create links?.................................................................................................72
Pathname abbreviations .......................................................................................73
Shell Meta-characters...........................................................................................74
* - Asterisk...........................................................................................................75
?- Question Mark..................................................................................................76
[ ] Square Brackets............................................................................................77
` ` - Backticks.......................................................................................................78
Quoting in Shells [ ]........................................................................................79
Escaping in shells [ \ ]..........................................................................................80
Getting Help.............................................................................................................81
Getting help : man................................................................................................82
man : Output.........................................................................................................85
man : example.......................................................................................................88
man page sections.................................................................................................92
manual entries in other sections...........................................................................93
Other useful man options......................................................................................94
Google..................................................................................................................95
Google: Example..................................................................................................96
SunSolve...............................................................................................................97
File Types and Listing .............................................................................................98
Varieties of Files ..................................................................................................99
What sort of file [file].........................................................................................100
Strings in a file [strings].....................................................................................101
Contents of a file [cat]........................................................................................102
Listing contents of files [more]..........................................................................103
Listing contents of files [less].............................................................................104
Top few lines [head]...........................................................................................105
Bottom few lines [tail]........................................................................................106
Fields in a file [cut].............................................................................................107
Counting things [wc]..........................................................................................108
Finding the differences [diff]..............................................................................109
Differences of 3 files [diff3]...............................................................................110
Sorting files and output [sort].............................................................................111
Handling duplicate lines [uniq]..........................................................................112
Redirection.............................................................................................................113
Why ?.................................................................................................................114
Standard Files.....................................................................................................115
Redirection : < > 2> ...........................................................................................116
Redirection : >>..................................................................................................117
Redirection : Merging ........................................................................................118
Redirection: Multiple redirections......................................................................119
Redirection : Examples.......................................................................................120
More redirection: File creation...........................................................................121
Pipelines : |.........................................................................................................122
Pipelines : examples...........................................................................................123
Named Pipes.......................................................................................................124
File Security............................................................................................................125
Security Overview..............................................................................................126

Permissions.........................................................................................................127
Finding the permissions [ls l]...........................................................................128
Types of users.....................................................................................................129
Access Permissions............................................................................................130
How is access decided ?.....................................................................................131
Controlling access [chmod]................................................................................132
Controlling access [chmod]: The modes............................................................133
Octal Modes.......................................................................................................134
Octal Modes : example.......................................................................................135
User ID and Group ID........................................................................................136
Real & Effective User/Group ID........................................................................137
Who am I [id].....................................................................................................138
Ownership..........................................................................................................139
Changing Ownership: chown and chgrp............................................................140
setuid..................................................................................................................141
setgid..................................................................................................................142
setuid and setgid with directories.......................................................................143
Sticky bit.............................................................................................................144
Default Permissions [umask]..............................................................................145
Searching and Translating......................................................................................147
Why?..................................................................................................................148
find : Finding files..............................................................................................149
find : Conditions.................................................................................................150
find : Examples...................................................................................................151
Regular Expressions...........................................................................................152
Regular Expressions : Meta-characters..............................................................153
Regular Expressions: Character Classes.............................................................154
Regular Expressions : Two warnings.................................................................155
Regular Expressions : Examples........................................................................156
Translate [tr].......................................................................................................157
Finding things [grep]..........................................................................................158
Stream Editor [sed].............................................................................................159
sed : Substitution Examples................................................................................160
sed.......................................................................................................................164
Who is using the system ? [who]........................................................................165
Quicker who [w].................................................................................................166
Find a spelling [look]..........................................................................................167
Processes and Control.............................................................................................168
Process Overview...............................................................................................169
Generating Processes..........................................................................................170
Understanding Processes....................................................................................172
Displaying the Processes [ps].............................................................................173
Displaying the Processes [prstat]........................................................................174
prstat : column headings.....................................................................................175
Managing Processes...........................................................................................177
Background a command [ & ]............................................................................178
nohup..................................................................................................................179
Listing the jobs [jobs].........................................................................................180
Foreground a command [fg]...............................................................................181
Killing processes [kill].......................................................................................182
Job Management Commands.............................................................................183
/proc....................................................................................................................184

vi: Visual Editor.....................................................................................................185

The Visual Editor...............................................................................................186
Modal Editor ......................................................................................................187
Editing a file.......................................................................................................188
Leaving vi...........................................................................................................189
Entering Text......................................................................................................190
Insert / Replace mode ........................................................................................191
Command Mode.................................................................................................192
Cursor Modes.....................................................................................................193
Moving around in vi...........................................................................................194
Deleting Text......................................................................................................195
Searching in vi....................................................................................................196
Replacing Text....................................................................................................197
More text replacement........................................................................................198
Substituting text .................................................................................................199
Coping text.........................................................................................................200
Retrieving text....................................................................................................201
Miscellaneous tricks...........................................................................................202
ex commands......................................................................................................203
! (bang)...............................................................................................................204
set commands.....................................................................................................205
Typical Set Commands.......................................................................................206
.exrc : making set commands stick.....................................................................207
Final Thoughts....................................................................................................208
Shells and Variables...............................................................................................209
What is an Environmental Variable ?.................................................................210
Where do they come from ?...............................................................................211
Example variables : Which shell?......................................................................212
Example variables : Process ID [$$]..................................................................213
Example Variables : Exit Status [$?]..................................................................214
Creating local variables......................................................................................215
Removing environmental variables....................................................................216
Making variables global [export].......................................................................217
Displaying the environment [set] [env]..............................................................218
Search paths [PATH]..........................................................................................219
Searching the PATH [which].............................................................................220
Making the environment stick............................................................................221
Initialisation file sequences................................................................................222
More about Shells...................................................................................................223
Quoting in Shells [ ]....................................................................................224
Escaping in shells [\]..........................................................................................225
Custom Prompts.................................................................................................226
Aliases [alias].....................................................................................................227
Removing Aliases [unalias]................................................................................228
Command recall [history]...................................................................................229
Command line editing: the vi way......................................................................230
Command line editing: the emacs way...............................................................231
Shell Scripting........................................................................................................232
What is a command ?.........................................................................................233
Shell Scripts........................................................................................................234
Flow control.......................................................................................................235
Testing conditions [test].....................................................................................236

Flow control: if...................................................................................................237

Flow control: else...............................................................................................238
Flow control: elif (i.e.else if)..........................................................................239
Flow control: nested if statements......................................................................240
Flow control: while / until..................................................................................241
Flow control: while / until examples..................................................................242
Flow control: for.................................................................................................243
Positional Parameters.........................................................................................244
Positional Parameters: continued.......................................................................245
Networking.............................................................................................................246
Networking and Distributed Systems.................................................................247
Basic Network Services......................................................................................248
Distributed File Systems.....................................................................................249
Distributed Window Systems.............................................................................250
Protocol Layers...................................................................................................251
The OSI 7 Layer Model......................................................................................252
The TCP/IP Protocol Stack................................................................................253
Concepts.............................................................................................................254
Connectionless protocols....................................................................................255
Connection-oriented protocols...........................................................................256
Ports + Sockets...................................................................................................257
Important Port Numbers.....................................................................................258
UDP - Fundamentals..........................................................................................259
UDP - Frame format...........................................................................................260
UDP - Issues.......................................................................................................261
TCP - Fundamentals...........................................................................................262
TCP - Frame format............................................................................................263
TCP - A Sample Session....................................................................................264
TCP - Options.....................................................................................................265
Name Services....................................................................................................266
Domain Name Service (DNS)............................................................................267
Domain Name Service: Examples......................................................................268
Structure of DNS name space.............................................................................269
LDAP..................................................................................................................270
/etc/nsswitch.conf...............................................................................................271
Connect to another host [telnet].........................................................................272
Fetching remote files [ftp]..................................................................................273
Inside ftp.............................................................................................................274
Secure remote connections [ssh]........................................................................275
Remote login without passwords [ssh]...............................................................276
Startup and shutdown.............................................................................................277
Boot Stages.........................................................................................................278
Firmware............................................................................................................279
POST output.......................................................................................................280
Open Boot PROM (OBP)...................................................................................282
OBP: boot command..........................................................................................283
Kernel Initialisation............................................................................................284
The /etc/system file.............................................................................................285
Boot output ........................................................................................................286
init and run levels...............................................................................................287
Predefined run levels for Solaris........................................................................288
The /etc/inittab file.............................................................................................289

Run Control Scripts............................................................................................291

A Typical Run Control Script.............................................................................292
The /etc/init.d directory......................................................................................293
Run level commands..........................................................................................294
Run level commands: shutdown.........................................................................295
Run level commands: halt..................................................................................296
Run level commands: reboot..............................................................................297
Run level commands: telinit...............................................................................298
Devices...................................................................................................................299
Local Devices.....................................................................................................300
Device Naming Conventions..............................................................................301
Device aliases [devalias]....................................................................................302
Partitions.............................................................................................................303
Why Partition?....................................................................................................304
Partition Tables...................................................................................................305
Listing partitions [prtvtoc]..................................................................................306
prtvtoc output.....................................................................................................307
Creating partitions [format]................................................................................308
The format Main Menu......................................................................................309
The partition Menu.............................................................................................310
Partitioning a disk using modify.........................................................................311
File Systems.......................................................................................................313
Network File Systems.........................................................................................314
Media errors.......................................................................................................315
Checking and fixing file systems [fsck].............................................................316
Backup: Principles..............................................................................................317
Backup Commands: tar......................................................................................319
Backup Commands: tar examples......................................................................320
Backup Commands: cpio....................................................................................321
Backup Commands: cpio examples....................................................................322
Backup Commands: dd.......................................................................................323
Backup Commands: mt......................................................................................324

Unix History and Lore

Unix History and Lore

The wonders of the ages assembled
for you edification, education, and
enjoyment for a price

P.T. Barnum

Introduction

Introduction
What is Unix?
Distributed Computing with Unix
Why Unix?
Unix Philosophy
Unix Major Features
Unix Lore
A Potted History
Unix Flavours
The Evolution of Unix

What is Unix?

What is Unix?
Multi-user, multi-process, multi-access
operating system

Unix
Allows users to run programs, manage their
own files and use devices

Unix is a multi-user, multi-process, multi-access operating system. This means

that it can support multiple simultaneous users, each executing multiple programs.
Unix provides an operating environment for users to run programs, manage files,
accessing devices, communicate with each other and coordinate their activities.
Unix is commonly used in networking environments, allowing data and resources to
be shared amongst the connected machines.

Distributed Computing with Unix

Distributed Computing with Unix

Users share resources around a distributed
network of machines
multiplexer / hub
wor kstations

terminator
repeater

ethernet segment
router

transceiver

Distributed environments consist of machines interconnected through some

networking mechanism. Machines are able to access each others disks, peripherals
and CPUs.
Individual users connected to a workstation have the resources of the entire network
at their disposal. Access to the network is often transparent; for example, an
application may physically reside on one machine, be executed on a users local
machine and make use of configuration and data files from a third machine. A
distributed network of computers and devices is usually presented as an integrated,
homogenous, environment to users.

Aims

Aims
Design Goal
... to create a computing environment where they themselves
(the staff of CSRG) could comfor tably and effectively pursue
their ow n work - programming research.

not so much goals in early design

OS for programmer, not for production

- simple and easy (not performance and functionality)

- interactive (no batch processing)
C form the original programmers preference

- easy porting -> wide spread of Unix

small hardware platform

- entails elegance

Why Unix?

Why Unix?
Highly portable, extensible & configurable
- written largely in C
- modular kernel and command set
- configuration soft coded
An open operating system
- vendor independent
Relatively standardised
- IEEE POSIX, X-Open XPG
- Open Software Foundation
Provides a productive development environment
- software tools
- file processing tools
Over 90% of the Unix operating system is written in the C high-level programming
language. This makes the system relatively easy to port to new hardware.
Unix is based on a few simple concepts and is structured in a highly modular
fashion. It has been able to evolve as new technologies become available, but has
remained relatively standardised. Such groups as the IEEE POSIX committee, the
X/Open Forum and the Open Software Foundation (and previously Unix
International) have strived to keep Unix developments open and standardised.
Unix is not tied to any particular vendor, being perhaps the first true open operating
system. Implementations of the system exist on some of the smallest machines in
the world (such as PCs) and also on the biggest machines (such as supercomputers).

Unix Philosophy

Unix Philosophy
Write programs that do one thing and do it well

Write programs to work together

Write programs to handle character streams,
because this is the universal interface
Unix has often been described as a toolbox. Rather than being an entity in itself, it
is a collection of tools which work together. The notion of command pipelines, the
ability to feed the output of one command directly into the input of another, led to
this philosophy.
Doug McIIroy, responsible for writing pipes, diff (and reportedly herding cats) was
one of the first to articulate the above philosophy.
Writing programs to do one thing and do it well was a departure from other systems
which tended to bundle heaps of functionality into one place. As a consequence,
such software was often large, expensive to run, and difficult to adapt to changing
requirements.
The problem with writing many simple programs comes when it is necessary to
achieve a complex task. The programs must be arranged to work together. Unix
achieves this through command pipelines, because most of the utility programs in
Unix adhere to the universal interface of the character stream. That is, commands
produce as their output and receive as their input a stream of characters. Such
notions as data or record format (and ideas of fixed and padded records) are largely
meaningless.

Major Features

Major Features
Simple, powerful, user interface
Complex commands are made
from simple ones
Hierarchical file system
Consistent file format, the byte stream
Simple, consistent, peripheral interface
Hides machine architecture from user

Unix provides hundreds of commands each designed to do one thing well. Through
a Unix shell (command line interpreter) collections of such commands are
combined to perform complex tasks.
In Unix, files on disk, devices and the input and output of running programs are
considered files. All physical devices have filenames, and behave as ordinary files.
The fundamental component of information in Unix is the byte stream. It allows
files, devices and even programs to be used interchangeably as the source or
destination of data; and thus allows the underlying machine architecture to be
hidden from the user.

Unix Lore

Unix Lore
Intensely loyal defenders acclaim

- extremely rich and powerful command set

- clean and terse command syntax
- ability to do things in beautiful and simple ways

Intensely critical detractors profane

- terse syntax and strange command names

- complex administration
- unfriendly to novice users

Unix is an old operating system (though it has changed considerably from its earlier
implementations). It started life as a support environment for a computer game,
and graduated into a teaching and research platform for Universities. As a
consequence, it has acquirednment, to others (especially novice users) the sheer size
and complexity of the command set make it difficult to use.

Potted History

Potted History
1965
1965Bell,GEC
Bell,GECand
andMIT
MITstart
startMULTICS
MULTICS
1969
1969Space
SpaceTravel
Travelconceived
conceived
Unix
Unixbuilt
builton
onPDP-7
PDP-7
1973
1973Unix
Unixcoded
codedininCC
1981
1981BSD
BSD4.1
4.1
1982
1982Unix
UnixSystem
SystemIIIIII
AT&T
AT&Tcombined
combinedseveral
severalvariants
variants
1983
1983UNIX
UNIXSystem
SystemVV
System
SystemIV
IVwas
wasan
aninternal
internalversion
version

Unix started life as the support environment for a Space Traveller game developed
at Bell Laboratories. It was originally coded in PDP assembler, some parts were
developed in B (an interpreted language influenced by BCPL) and later the entire
system was recoded in C. C was developed by Dennis Ritchie for Unix to
overcome the performance limitations of B. Unix was named by Brian Kernighan
as a pun on an earlier system called Multics.
Once Unix had become a stable product within Bell Laboratories, consideration was
given to selling it. However, due to a Consent Decree Bell had signed with the US
Federal government in 1956, it was forbidden to market computer products.
Instead, Bell laboratories (specifically, the Unix Systems Group of AT&T) gave the
system to Universities for educational purposes. Research at the University of
California at Berkeley lead to the development of a variant of the Unix system.
Most commercial flavours of Unix are based upon AT&T system V Unix, or BSD
(Berkeley Software Distribution) 4.x Unix.

Potted History
1983
1983BSD
BSD4.2
4.2
1984
1984SunOS
SunOSderived
derivedfrom
fromBSD
BSD
1985
1985BSD
BSD4.3
4.3
1989
1989SunOS
SunOS44(Solaris
(SolarisI)I)

1991
1991Convergence
Convergenceon
onSVR4
SVR4

1992
1992Solaris
SolarisIIII(SVR4)
(SVR4)

SunOS is Sun Microsystems flavour of Unix. It is based largely upon BSD Unix,
with NFS and NIS network extensions to allow files to be shared and managed
around a network.
In the interest of standardisation, Sun Microsystems, together with a number of
other major workstation manufacturers, have converged on SVR4 Unix. System V
Release 4 incorporates many of the features of earlier System V versions of Unix,
in addition to many other facilities. Suns SVR4 product is Solaris II, and all
preceding SunOS products are now referred to as Solaris I.

Unix Flavours

Unix Flavours
Workstation Unixes
Sun Microsystems
Sun Microsystems
Silicon Graphics
IBM
Hewlett-Packard

SunOS-4 (Solaris I)
SunOS-5 (Solaris II)
AIX-3
HP- UX - 8

Personal Computer Unixes

Santa Cruz Operation SCO Unix V
IBM
AIX-1
Interactive
Public Domain

Unix
Linux

BSD + SVR3
SVR4
IRIX 4 SVR4
SVR3 + ?
SVR3 + BSD
SVR4
SVR3
SVR4
SVR4

Mini and Super Computer Unixes

DEC
Meiko

Ultrix
Meikos

BSD + DECnet
SVR3

The popularity of Unix is demonstrated by the variety of platforms on which it is

available. From PCs to workstations, and mini-computers to super-computers,
Unix provides a standard operating system.
The various flavours of Unix descend either from AT&T Unix or from Berkeley
Unix. AT&T System V has been particularly influential on the development of PC
based Unix systems, whilst BSD Unix (Berkeley Software Distribution) has
influenced mini and workstation Unixes.
One of Unixes strengths is that it has been in the public domain for much of its life.
Consequently, it has evolved to take advantage of new technology and has become
an integral part of many embedded, control and general purpose systems. This
strength is also a weakness, however, since it has allowed many diverse versions of
Unix to evolve. Standards activities promoted by the X-Open Forum, the IEEE
POSIX committees and the Open Software Foundation have helped to pull the
systems together. Currently, most of the important versions of Unix are converging
towards System V Release 4 (SVR4).

This page intentionally left blank

The Evolution of Unix

1965 Multics
A joint project started between AT&T Bell Laboratories, project MAC of MIT and General Electric. The aim
was to develop MULTICS (MULTIplexed Information and Computing System) an interactive, multi-user,
multi-access operating system.
1969 Unix begins ...
Bell Labs pull out of the MULTICS project because it failed to meet its design goals. Ken Thompson develops
the initial basis for Unix on a discarded PDP-7, including a filesystem, basic process model and an interactive
shell. The system was developed to support a Space Travel program, which simulates the motion of planetary
bodies. The system was later named Unics and then Unix, as a pun on the original MULTICS project by Brian
Kernighan.
1971 Unix ported to PDP-11
The first real project supported by Unix was to provide a text processing system for the patent department at
Bell Laboratories. In order to achieve this, the system was ported to a PDP-11, providing 16K for the system,
8K for user programs, a disk of 512K bytes and a limit of 64K bytes per file.
Ken Thompson, in an attempt to implement a FORTRAN compiler for the new system, came up with the
language B. This was an interpreted language used in the development of some Unix tools. B was developed
from BCPL, and named after the first initial.
1972 C Developed
Interpreted languages tend to realise poor performance. In an attempt to overcome the performance limitations
of B, Dennis Ritchie developed the now popular language C. C was named from the second letter of BCPL. It
provides for declaration of data types, definition of data structures, low level machine access and allows the
direct generation of machine code.
1973 Unix coded in C
A major step in the acceptance of Unix was its recoding in C. This allowed it to be easily ported and extended,
and lead to the formation of the Unix Systems Group to provide internal support within Bell Laboratories.
1974 Unix distributed to universities
AT&T were prohibited from marketing computer products because of a 1956 Consent Decree it had signed with
the US government. However, it provided Unix to universities who requested it for educational purposes.
AT&T neither advertised, marketed nor supported the system in accordance with the terms of the Consent
Decree. As a consequence, the system was distributed with all of the source code. This version of Unix was
known as 5th edition.

This page intentionally left blank

1975 Thompson attends UCB

In 1975 Ken Thompson took a sabbatical to the University of California at Berkeley (UCB). This marked the
start of Unix development at UCB within the Computer Systems Research Group (CSRG). Unix software
released from Berkeley is released in Berkeley Software Distributions; for example, 4.3 BSD.
1977 Unix ported to non PDP machine
By 1977 the number of Unix sites had grown to about 500, of which about one quarterty of Unix began to grow,
as marked by the first port of Unix to a non PDP machine and the establishment of the first Value Added
Reseller (VAR) of a Unix system. The current version was the 6th edition.
1978 7th Edition Unix
The first Unix system with portability as a specific goal was Unix Time-Sharing System, Seventh Edition (V7).
The system ran on the PDP-11, the Interdata 8/32 and had a DEC VAX 11/780 port called Unix/32V. The
VAX version provided 4Gb virtual address space.
After the release of V7 the external distribution of Unix systems was turned over to the Unix Support Group
(USG).
1979 3 BSD
The first significant Berkeley Unix work was an enhanced version of VAX Unix/32V. In this system, known as
3 BSD, support for virtual memory, demand paging and page replacement was added. The large virtual
memory of 3 BSD was motivated by large program development at Berkeley.
1980 4 BSD
4 BSD was produced between 1979 and 1980 for the American Defense Advanced Research Projects Agency
(DARPA). An important goal of the projecttly general to allow communication among wide and local area
networks.
4 BSD was the operating system of choice for VAXes from when it first became available until the release of
System III. Due to licensing limitations, organisations would buy a 32V licence from AT&T, and then order 4
BSD from Berkeley. Eventually, DEC produced ULTRIX, its own version of 4 BSD (which included support
for DECs proprietary network DECNET) and sold this directly with VAX machines as an alternative to VMS.
1981 /usr/group formed
The first independent organisation for the standardisation of Unix systems was introduced. Originally known
as /usr/group, the organisation was later renamed UniForum.
1982 Unix System III
In the period between 1977 and 1982 several variants of the AT&T Unix system were combined to produce
Unix System III. The system included some features from BSD Unix, including terminal independent support
for screen based applications (curses and termcap) and the C-shell.

This page intentionally left blank

1983 Unix System V & 4.2 BSD

AT&T Bell Laboratories further develop Unix System III and call the new product System V (System IV was
an internal evolutionary version). System V was released by the USG and for the first time official support for
the system was announced.
In addition, and of great significance to the future development of Unix, the US court-ordered divestiture of
Bell Operating Companies from AT&T left AT&T in a position to market System V aggressively.
The culmination of the original Berkeley DARPA Unix project was released as 4.2 BSD.
Sun Microsystems released SunOS 1.0, a version of Unix derived from BSD 4.2. Suns reputation is based
around networked workstations and its enhancements to BSD were to support this kind of environment. Sun
introduced a window based user interface, called Sun Tools, and a distributed Network File System (NFS).
1984 Unix System V Release 2
After a transformation of the USG into the Unix System Development Laboratory (USL) Unix System V
Release 2 (SVR2) was released. The system introduced paging and shared memory.
Microsoft produced Xenix, the first significant, commercially licensed, derivative of AT&Ts Unix Version 7.
Xenix was designed to run on PC architecture machines, but was also ported to larger machines. The system
was eventually acquired by the Santa Cruz Operation (SCO).
By this time there were in excess of 100000 Unix installations around the world running on machines from
microprocessors to mainframes. No other operating system has ever been so prolific.
The US /usr/group published a document defining the Unix operating system. Although the report was too
brief to provide a complete description of the system, it did form the basis of the future IEEE POSIX
standardisation work.
1985 System V definition and verification
As several manufacturers bundled Unix with there systems, AT&T attempted to enforcee Definition (SVID)
was produced to unambiguously define System Vs call interface, and the System V Verification Suite (SVVS)
was produced to check that a Unix system was compliant.
Microsoft released Xenix 3, an extension of their previous product based on System III and 4.2 BSD.
1986 4.3 BSD
Berkeley software distribution 4.3 was released to primarily fix bugs in the 4.2 BSD product. In particular,
there were problems with the TCP protocol implementation and performance problems with new features such
as symbolic links.
The Santa Cruz Operation (SCO) took over distribution of Xenix from Microsoftily a European initiative to
define a Common Applications Environment (CAE) independent of hardware platform. The group consists of a
consortium of manufacturers, including AT&T, DEC, ICL and Sun.

This page intentionally left blank

1987 Unix System V Release 3

The Unix System Development Laboratory was succeeded by AT&T Information Systems (ATTIS) which then
distributed Unix System V Release 3 (SVR3) The system included the STREAMS generalised network
interface, shared libraries and a mechanism for interprocess communication.
IBM released the Advanced Interactive Executive (AIX-1) as an enhanced version of Unix for its 6150 PC-RT
range of machines.
The X-Open group produced the X-Open Portability Guide (XPG) to promote the portability of applications
between diverse Unix implementations. The guide provides information relating to kernel and system
interfaces, and utility programs.
In late 1987 AT&T announced that it had purchased a sizeable portion of Sun Microsystems. Following this
Sun announced that its next operating system would not be a further extension of SunOS (BSD derived) but
would be based on Unix System V Release 4 (now known as Solaris II). This raised some degree of horror in
other parts of the Unix community which felt that this special relationship between AT&T and Sun would
threaten their market.
1988 IEEE POSIX & OSF
The IEEE created the POSIX standards as an independent family of standards for software and hardware
manufacturers. The POS in POSIX stands for Portable Operating System, the IX is borrowed from Unix. The
standards are produced by committeeerating system interface functions, interactive shells, basic commands and
real-time processing.
Sun Microsystems released SunOS 4.0, a major version of the SunOS family. This was the last major release
based on BSD, since SunOS 5 is derived from SVR4.
In May 1988 the Open Software Foundation (OSF) was formed. This is an independent organisation for the
further development and provision of computing environments, based on current standards and technologies.
Perhaps more significantly for the members of OSF, the aim was to produce a standard Unix environment
independent of AT&T licensing. The group of companies pulled together to make OSF included Apollo, DEC,
Hewlett-Packard, Honeywell-Bull, MIPS, NCR, Silicon Graphics and IBM. The formation of OSF was the
direct result of AT&Ts procurement of a stake in Sun Microsystems, and the consequential dangers to their
market. At the time, it was noted that the OSF had the distinction of getting IBM and DEC into the same bed!
As a consequenceun formed a counter-consortium called Unix International, dedicated to SVR4.
The American National Standards Institute (ANSI) completed the standardisation process for C. However,
since the committee was in progress for five years, most of the features of the standard were already supported
by modern compilers.
1989 OSF Motif
This year saw the growth of Unix International and the Open Software Foundation; companies and private
members joined both. By the end of the year, OSF released its Motif user interface. The distinction between UI
and OSF was, however, relatively small and even at this stage there was talk of a merger.

This page intentionally left blank

1990 AIX-3 for the RS/6000

IBM released AIX-3 for their RS/6000 Unix workstation. AIX-3 is a radical rewrite of the earlier
AIX-1 product. Whilst maintaining some conformance to its System V ancestor, the system
includes a number of IBM specific extensions.
1991 HP-UX Version 8.0
Hewlett-Packard released HP-UX version 8.0. The system is based on SVR3 with extensions from
4.2 BSD and 4.3 BSD. Emphasising standardisation, the system is compliant with SVID2,
POSIX and relevant XPG interfaces.
1992 Sun released Solaris II
The first release of Solaris II was distributed by Suns software company, SunSoft. The product is
also known as SunOS 5.0, and is Suns first implementation of SVR4 (and marks, therefore, Suns
departure from BSD Unix).
The wrangling between the OSF and UI decayed, and UI was disbanded. Moreover, Sun
(formerly of UI) adopted OSFs Motif as its user interface for Solaris.
OSF continued development of DCE, the Distributed Computing Environment. DCE represents a
significant effort to link diverse computers into a single, productive network. DCE systems
should be able to: support the creation of a single network filesystem (similar to NFS), make
network services transparently accessible, balance computing load around a network, provide
network management facilities, and provide a flexible network security mechanism.
1993 Common Open Software Environment
A proposal by leading hardware and software companies for a common desktop environment was
presented to X-Open for incorporation into the X-Open Portability Guide (XPG). The Common
Open Software Environment (COSE) should provide a consistent computing experience, with a
single set of programming interfaces, for HP, IBM, SCO, SunSoft and AT&T platforms.
Sun continued work on DOE, the Distributed Objects Everywhere project. DOE is indicative of
many object-oriented operating system projects currently underway. In such systems, the entities
that make up a distributed computing environment are considered to be autonomous objects, and
the networked system is an amalgam of interacting objects.
1994 SunSoft & NeXT ...
SunSoft and NeXT announced a series of agreements designed to establish a comprehensive set of
object-oriented application development standards. This is another significant thread in the
development of distributed object-oriented environments, which is becoming the distinctive mark
of the new generation of operating systems.

Key Concepts

Key Concepts
Great is the art of beginning,
but greater is the art of ending
Lazurus Long

The Business of Computing

The Business of Computing

Computer programs process data
input
data

processing

output
data

files provide input and receive output

processes manipulate data to produce new data

There are only two entities in Unix

The business of computer programs is to process data. To read data from some
source, apply some computation, and generate result data. This model is supported
directly by Unix.
In Unix there are only two entities: files and processes. Files represent the data
being read or written, processes are the active entities reading, processing and
writing data.

Files & Processes

Files & Processes

Everything is a process or a file
Files are passive entities

- streams of bytes stored on disks

- interfaces to devices
- the input and output streams of running programs

Processes are active entities instances

of running programs
- instructions for the CPU

Every process starts life as a file

- Unix commands are stored as program files on the disk

In Unix, everything is a process or a file. No other entities exist, not even disks,
printers, terminals or networks---in Unix all of these things appear as files. The
idea of using file names to represent devices saves introducing another concept.
When a devices file is read or written, Unix ensures that the interactions are
propagated to the particular device which the file represents.
Files are passive entities, unable in themselves to do anything. Processes are active
entities, in some sense they have life. Consider a human as a process, and a
suitcase as data. The suitcase cannot move itself, because it does not have life. A
human must be applied to the suitcase in order for it to move. Likewise a process is
applied to a file in order for it to be processed.
Unix is unusual compared with many operating systems in that process creation is
relatively inexpensive. As a consequence, each command is executed as a single
process. This differs from other systems which often run commands as procedure
or function calls within a central command process.
In Unix, each invocation of a command gives rise to a new process. The process is
created from the commands program file, executes and then dies. It is not unusual
for one command to be executed by one or more users simultaneously, giving rise
to many independent processes (instances) each performing the same task.
Note that the instructions as to what a process should do are stored in a
corresponding program file on disk. When a command is issued, the text of its
program file is used to make the process. The CPU interprets each instruction
within the process to carry out the work.

Organisation of Files

Organisation of Files
Files are organised as an inverted tree
The Unix File System consists
of a single root directory which
contains files.
Files may be data, programs
devices or directories.
This logical file system may be
composed of many physical
devices and networks.

Both process and file entities in Unix are organised as trees. The tree used to hold
files is called the Unix File System (UFS). The tree used to hold processes is
simply called the process tree.
Unix File System
The Unix File System is organised as an inverted tree; the root is at the top, and
branches and leaves in the form of directories and files grow down. A directory is a
special file which can hold other files. Since these files may themselves be
directories, a tree structure is formed.
Unix systems contain only one logical file system. The file system may span
multiple partitions and disks, cross networks and exist in multiple physical forms.
However, the illusion maintained by the operating system, is that the file system is
one, coherent, tree. As a user moves around the file system, Unix ensures that the
physical joins between disks or networks remain hidden.
Since there is only one file system, all users, programs and devices exist in the same
name space. By convention, programs and devices exist in their own subdirectories, and users exist in their own sub-directories.

Organisation of Processes

Organisation of Processes
Processes are organised into a process tree
All processes have a parent
process (save the first) and
may have child processes.
Each command is executed
as a new process, and is the
child of the process which
invoked it.

Unix does not discriminate between the execution of system related programs and
user programs. Both give rise to processes.
The first process in a Unix system is called init and it runs as part of the procedure
of making the machine ready for users. init creates child processes which setup the
machine and ultimately prompt the user to login. Once a user has logged in, a new
child process is created to enable the user to enter commands. This process is
called a shell.
The shell prompts the user to enter a command. For each command that the user
enters, the shell spawns a new processes. The shell is therefore the parent of these
processes. The shell is itself the child of init, since this gave rise to its creation
when the user logged in. Many Unix commands also give rise to child processes,
which may in turn give rise to new generations of processes. Thus a tree of
processes is seen to have been spawned, starting from the great grand parent of all
processes, init.

Login

Login
Multi-user OSs require users to login
- validate username
- associate with account

The login process establishes the users

- initial process (shell)
- initial directory (home directory)
- Identity (User and Group)

Users view Unix through

-

interactive shells
custom menus
windowing environments
applications

In order to use a Unix machine users must login. In this process the system
determines who the user is (the username), verifies this information (by requesting
a password), and then associates with the user their file and process resources.
Specifically, the user is associated with a sub-directory of the filesystem (their
home directory) and an initial process (usually an interactive shell).
Humans tend to use names to distinguish similar objects, machines tend to prefer
numbers. During the login process the system associates a UID (user identity
number corresponding to the username) and GID (group identity number
corresponding to the users default group) with the user. This is subsequently used
to label all files and processes created by the user.

Login File & Process

Login File & Process

Home Directory and Login Shell
/

sh

home

fred

The login shell is a

command line interpreter,
invoking commands on
behalf of the user

init

The home directory is

a sub-directory which
contains files which
belong to the user

cat file

Once a user has successfully logged-in to the system, the user is placed in their
home directory and is given an initial shell from which they may invoke commands.
The home directory
The home directory is simply a sub-directory within the single, logical tree structure
of the Unix file system. Since this structure may be composed of both remote and
local disks, it is likely that the actual physical location of the home directory resides
over the network on a carefully managed file server machine.
Each user usually has their own home directory and uses this to store personal files,
such as programs, letters and data. Home directories may themselves be broken
down into sub-directories, to help organise the users files.
The home directory is said to be the top of the users file tree. Note, however, the
user may still go to any other part of the tree provided they have suitable access
rights.
The Login Shell
The login shell is a process executed on behalf of the user when they login. The
command giving rise to the process may be any Unix command. Often it is an
interactive shell, allowing users to enter their own command. Sometimes it is a
specific application, locking the user within the application and hiding Unix itself.
The login shell is said to be the top of the users process tree. The user may logout
by terminating this process.

What are Users?

What are Users?

Users are the owners of files and processes
process

process

fred

george
file

file

Users are not first class entities within Unix, only files and processes can claim this
status. Users are simply attributes of files and processes. In Unix, every file and
every processes must be owned by someone and exist in a group.
When users first login their initial process (the shell) and file (their home directory)
belongs to them. Every subsequent file or process created by a user is stamped with
the users identity (UID) and the users current group identity (GID). The UID is
usually unique and has a one-to-one mapping with the users username; the GID is
shared
by users
together.
The GID
provides
In
Unixworking
everything
carries
a UID
anda means
GID by which users can
gain joint access to shared files and commands.

Unix Structure

Unix Structure
more

Unix has a highly modular,

layered, structure

cat

ksh

ls

Utilities are layered around

the kernel

kernel

vi

sh
sort

grep

The structure of the Unix system is different from most other large operating
systems. At its centre is a core of functionality known as the kernel. This provides
the system resources, such as the management of process and file trees.
Around the kernel are layered the programs which make up a users perspective of
the operating system. Hundreds of individual commands use the kernel in order to
achieve their tasks. The commands physically reside in the filesystem as program
files. However, once executed they are loaded into the systems memory and
executed as processes.
In the above picture, some of the standard commands are shown around the kernel.
The commands marked sh and ksh are interactive user shells. They are usually
automatically executed when a user logs into the machine and invoke commands on
the users behalf. Its interesting to note, that the shells in themselves are no
different from other programs. Although they form the layer between the user and
the kernel, they reside in the filesystem and are executed as processes.

Kernel Responsibilities

Kernel Responsibilities
sh

who

vi
kernel

The kernel is responsible

for
process management
memory management
filesystem management
device management

The kernel is responsible for providing the resources needed by the programs
wishing to run. There are primarily four resources which the kernel manages.
Process Management
Unix systems are multi-user, multi-process environments. This means that at any
point in time there may be several processes running (between 15 and 60 is not
unusual). Most of the hardware on which Unix is run only has one processor. The
process management part of the kernel is responsible for scheduling each of the
processes
able to calls
run onto
the CPU.hooks
This isinto
referred
as time-slicing or multiSystem
provide
thetokernel
tasking. Provided the kernel can switch from one process to another fast enough
(called a context switch), the illusion to the user is that all of the processes are
running simultaneously.
File System Management
The notion of a tree structure is maintained by the kernel. It allows users to manage
parts of the tree, hides the division between multiple disks and networks, and
protects one users files from another.
Memory & Device Management
Unix employs a virtual memory management policy in which the amount of
memory available to processes is greater than is physically available in RAM. This
is achieved by swapping (or paging) processes from RAM to disk (the swap region
on the disk) when the process is not needed, and swapping it back when it is.
Device Management refers to the maintenance of the file interface to devices.
When users interact with device files, the file like interactions are mapped by the
kernel onto the corresponding device.

Getting Started: Login

Getting Started: Login

My voice is my passport,
verify me

V. Brandis

CDE Login Screen

CDE Login Screen

Hostname of remote machine

Menu of login options

CDE, the Common Desktop Environment, is a Graphical User Interface, or GUI,
which is provided as standard across a number of different Unix platforms.
It was designed to provide end users with a consistent graphical user interface
across workstations and PCs.
The project was intended to:
Reduce learning time by providing the same appearance and behavior across
multiple operating systems.
Increase productivity by helping system administrators and end users
customize the desktop environment to fit individual work styles and
preferences.
Make learning easier by providing a consistent, rich, and easily accessible
context-sensitive on-line help system for help whenever and wherever the
user needs it.
Provide a common set of desktop and application development tools.
Ease the porting of many existing X Window applications to a new
environment. Applications should be easy to run across many different
platforms thus reducing the costs of moving to a new environment and
helping to protect investments in software.
Core components of the environment include:
A login manager (shown in slide)
A graphical login screen and manages user access to the system.
A file manager
An on screen graphical file representation where users can directly
manipulate icons associated with files to organize the file system
and launch applications.
An application manager

The application manager is similar to the file manager except that it

is intended to be a user specific list of files.
A session manager
Users can easily customize their environment.
The CDE window manager
The control mechanism for the visual user interface, or desktop, of a
session.
The CDE window manager includes a FrontPanel and a workspace
manager.
A user can manage all aspects of a session (except the initial
login) through objects on the FrontPanel.
An inter-application messaging system (similar to the Windows
Clipboard)
This aims to provide facilitates for the seamless interaction between
applications.
A desktop tool set
A comprehensive set of productivity tools including multimediaenabled mail, text editor, calendar, clock and icon editor, are
provided with the CDE.
Application development tools
A comprehensive set of development tools including debuggers,
application manager, application (Motif GUI) builder are provided
with the CDE.
Application integration components
Aplications written on any X Window system or toolkit should be
easy to integrate with the CDE tools provided.

Command Line Login Screen

Command Line Login Screen

Trying trinity.matrix.com...
Connected to 10.1.5.1.
Escape character is '^]'.
SunOS 5.9
login:

The CDE command line login screen is only accessible from the CDE login
managers options menu, when using a local login session.
Command line login can be achieved remotely using tools such as telnet and ssh.

Usernames

Usernames
Should be at least 6 characters long
Only the first 8 characters are significant
Case sensitive as in all Unix
Spaces are not allowed
Control characters and punctuation are permitted
Most people avoid them
Control-s and control-g are not permitted

Best login names contain both characters and digits

A login must exist on the system prior to attempting to log in as that user
Only the Super-User may issue logins

Valid Usernames

Valid Usernames
Valid Entries

Invalid Entries

cowboy

the_black_marauder

>8 characters

marauder

john smith

Contains space

johnny

sarah^g

Contains ctrl-g

johnnyQ

sam

Too short

cee3po
MyLoGiN
its_me

Best to choose a system and stick to it, e.g.

first name + first letter of family name
department code + initials + serial number
initials and year of entry
Think of a way of dealing with conflicting names before they occur

Good Passwords

Good Passwords
Must be at least 6 characters long
Only the first eight characters are significant
Case sensitive
Spaces, punctuation, and control characters are
permitted
(but not control-s and control-g)
Should be different than login name
Must differ from the old password
Before picking a password, it is important to consider security. A good password is easy for you to
remember and hard for someone else to guess.
"since the password protects your work and the entire system from unauthorized and careless
users, and since it is relatively easy for other users (all throughout the networked world no less!) to
move through Unix systems, you should carefully guard your password."
Passwords should be at least six characters long. They can be up to eight characters long, including
numbers and special characters. Using eight characters makes your password more secure. Try not
to use a word in a dictionary or a proper noun because they are so easy to guess just by going
through the dictionary. Personal information, like your social security number, phone number or
birthday is also easy to guess (as you know from all those spy movies you have been seeing).
In particular, do not use your username. Mix letters, numbers, and punctuation. Mix lower and
upper case letters, too. A good way of getting a secure password is to think of a phrase and then
take the initial letters of the words in the phrase, randomly capitalizing and adding punctuation.
License plate combinations also work well, however, dont use your own. Keep several passwords
in mind, so you can change your password regularly (every few months or any time you think
someone else knows it).

Passwords

Passwords
Any user may change their own password with
the passwd command
Super-User cannot decode your password
Though they can be brute force decoded using
crack

If old password is forgotten, a new one must be

issued
Passwords never appear on the screen

Keep in mind that your password is your own. No one else should know it. Do not
share your account with other users by giving your password to them. Depending
on the conditions under which your account was given to you, sharing it may be
like sharing your driver's license, it is often grounds for revocation. Please do not do
it.
Keep your password secure. Do not keep it in a computer file, send it by email,
write it on a post-it note or your white board.

Changing Passwords [passwd]

Changing Passwords [passwd]

trinity$ passwd
passwd: Changing password for morpheus
New Password: password
Re-enter new Password: password
passwd: password successfully changed for morpheus

System prompts for old password

System then prompts for new password
System then prompts for new again, to confirm
If passwords do not match,
system prompt for new password again.
Take care when setting your password. You will not be shown your password as
you type it in.
The system will perform some basic checks on your password as defined by the
operating system vendor when you enter it. You may be warned or prevented from
changing your password, if your password does not meet these checks.

Login Failures

Login Failures
A login failure will generate a login error which
must be acknowledged
If login does not exist
Screen will be cleared and a new login screen
generated

If login does exist, but password is incorrect

Only the password entry is cleared
The cursor is positioned for a new password attempt.

After a certain number of tries, the entire screen

is cleared

Why login fails

Why login fails

Incorrect username or password
Frequently a login will look correct, but is not
- It may contain a leading space, which is difficult to see

Name service failure

Name ser vice pr ocess isnt running
Netw ork failure between login machine and name service
provider

/etc/nologin
File created by super-user to prevent non-root logins, e.g.
during maintenance

If login continues to fail after the second try

Contact the Super- User.

If usernames and passwords are managed by a central networked name service,

such as LDAP or NIS, it will not be possible to log in as a user to a machine that
uses this service when it is disconnected from the network.
The super-user may also set a machine to refuse logins from ordinary users when
maintenance work needs to be performed. This is done by creating an empty file in
the /etc directory, called nologin.

Interruptions

Interruptions
To stop a command from executing

Ctrlc

To stop things from scrolling

Ctrls

To restart things scrolling

Ctrlq

To stop input and close a session

Ctrld

To erase the current line

Ctrlu

Ctrl-c terminates a process. Ctrl-d asks it to stop running in a more graceful

fashion.

Ending the sessions [logout]

Ending the sessions [logout]

$ logout
$ Ctrl-d (hold down the Ctrl key and D)
Sends the End-Of-File Character
$ exit
end the current shell
Click on the EXIT button on the Front Panel

/etc/passwd : home of identity

/etc/passwd : home of identity

/etc/passwd file
root:x:0:0:root:/root:/bin/sh
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:100:sync:/bin:/bin/sync
bill:x:1002:100:Bill_Smith,,,:/home/bill:/bin/ksh

Generic Format
login:x:uid:gid:gecos:home directory:shell

login
x
uid
gid
gecos
shell

the username
a place holder for the password, kept in /etc/shadow
a unique number to identify each user
a unique number to identify the users primary group
a comment field used to identify the user
the program executed on completion of the login

NAME
passwd - password file
SYNOPSIS
/etc/passwd
DESCRIPTION
The file /etc/passwd is a local source of information about
users' accounts. The password file can be used in conjunction with other password sources, such as the NIS maps
passwd.byname and passwd.bygid and the NIS+ table passwd.
Programs use the getpwnam(3C) routines to access this information.
Each passwd entry is a single line of the form:
username:password:uid:gid:gcos-field:home-dir:login-shell
Password is generally substituted by an x to show that the actual password is
found in the /etc/shadow file.

/etc/shadow : Password vault

/etc/shadow : Password vault

/etc/shadow file
root:zoJMzm8KNESF6:6445::::::
daemon:NP:6445::::::
bin:NP:6445::::::
bill:4sEixbwp7aoak:12925::::::

Generic format
username:password:lastchg:min:max:warn:inactive:expire:flag

username
password
lastchg
min
max
warn
inactive
expire
flag

The user's login name (UID)

A 13-character encrypted password for the user or NP
A number of days indicating when the password was last modified.
The minimum number of days required between password changes
The maximum number of days the password is valid
The number of days before expiry that the user is warned
The number of days of inactivity allowed for that user
An absolute date specifying when the login may no longer be used
Reserved for future use, set to zero. Currently not used

NAME
shadow - shadow password file
DESCRIPTION
/etc/shadow is an access-restricted ASCII system file that
stores users' encrypted passwords and related information.
The shadow file can be used in conjunction with other shadow
sources, including the NIS maps passwd.byname and
passwd.byuid and the NIS+ table passwd. Programs use the
getspnam(3C) routines to access this information.
The fields for each user entry are separated by colons. Each
user is separated from the next by a newline. Unlike the
/etc/passwd file, /etc/shadow does not have general read
permission.
Each entry in the shadow file has the form:
username:password:lastchg: min:max:warn: inactive:expire:flag

/etc/group: where groups live

/etc/group: where groups live

/etc/group file
root:x:0:
daemon:x:1:
bin:x:2:
sys:x:3:
adm:x:4:
tty:x:5:
admin:x:1000:morpheus,tank,cypher,neo

Generic format
group:x:gid:user-list

group
x
gid
user-list

:
:
:
:

name of group
password placeholder
a unique number to identify the group
list of users belonging to the group

NAME
group - group file
DESCRIPTION
The group file is a local source of group information. The
group file can be used in conjunction with other group
sources, including the NIS maps group.byname and
group.bygid and the NIS+ table group. Programs use the
getgrnam(3C) routines to access this information.
The group file contains a one-line entry for each group
recognized by the system, of the form:
groupname:password:gid:user-list
The password is generally replaced by an x

Summary

Summary
Usernames
6 characters or greater
8 characters are significant
Created in /etc/passwd or name service

Group
Primary gr oup is in /etc/passwd or name ser vice
All groups are in /etc/group or name ser vice

Login might fail

Bad username/password
/etc/nologin
name service failures.

File Hierarchy

File Hierarchy
Just because something
doesnt do what you planned
it to do, doesnt mean its
useless

T. Eddison

The Unix File system

The Unix File system

An organised way of storing files
The structure of the file system can be thought
of as an inverted tree

Unix employs a tree structure to store files. Starting from an initial top-level
directory (the root directory) sub-directories successively organise information into
categories, and then sub-catagories. There are no limits on the depth to which the
tree structure can grow.
Unix differs from other hierarchical file stores (such as those provided in DOS and
VMS) in that there is only one tree. The single tree structure hides multiple disks,
partitions and even the network when NFS (the Network File System) is employed.

Hierarchical Structure

Hierarchical Structure
Directories are files which hold information on
other files
Directories can be viewed as branches and files
as leaves
Since directories are just other files, they can
also be stored inside directories

The Unix file system is organised into a hierarchical tree structure in which
directories are branches and files leaves. The purpose of directories is to group
together related files. However, since files may themselves be directories, it
follows that directories may contain sub-directories.

Directory Paths

Directory Paths
/

mbox

A list of directorie s form s a path

News

report

format

text

style

letter

book

note

Absolute path names start from root (/)

/report/style/book
Relative path names start from the current directory
style/book
Path names describe routes through the file system. A relative path name is a route
from the current working directory; an absolute path name is the route from the top
of the file system.
Absolute path names begin with a leading slash and are unique. Relatively path are
not unique since they depend on the directory in which the path is specified.

File names

File names
Case sensitive
letters

A-Z or a-z

numbers

0-9

underscore _
period

comma

root

Other characters allowed but discouraged

There are no limits to the length of a file name.

What is a file?

What is a file?
A file consists of
an inode
data

The inode contains information about the file

file type, mode, ownership, creation date/time, etc.

pointers to physical location of data on device

Data is stored across a physical device

A file does not know its name

names are associated with inodes in directories

The file hierarchy is provided to offer an logical way of accessing data from
physical media.
Files consist of an inode and the actual data. Each file has an inode.
The inode contains information about the file. For Solaris, this is ususally:
the type of file (regular, directory, device, pipe, link, socket)
the mode of the file (read, write and execute permissions)
the number of hard links to the file
the user id of the file owner
the group id of the group to which the file belongs
dates and times of creation, last modification and last access
pointers to where the data is stored physically on the media
The inode does not contain the name of the file.
Other operating systems and file systems put different information in their inodes.

Varieties of Files

Varieties of Files
Ordinary Files
ASCII text or binary data, formatting left to user
Subclass of ordinary is hidden (name starts with .)

Directory
Directory provides a connection between the names
of the files and the files themselves. They impose a
structure on the file system

Device Files
Often located in the /dev directory
Writing to the device file transfers data to the device.
A directory is a file whose data is a list of file names and the locations of the inodes
that describe those files.
Device files are used to access hardware. They come in two types, block and
character.

Traversing the File System : cd

Traversing the File System : cd

Every directory contains two special directory
files
-.
-. .

current directory
parent director y

trinity% cd ../style

trinity% cd /report/style

trinity% pwd
/report/style

trinity% pwd
/report/style

trinity% cd ..
trinity% pwd
/report

trinity% cd /report/text
trinity% pwd
/report/text

trinity% cd
trinity$ pwd
/report/style

trinity% cd .
trinity% pwd
/report/text

The directories . and .. are convenient names for the current and parent
directories. .. allows path names to traverse back up through the tree.
Use the cd command to change directory and the pwd command to print the
current working directory. Note that cd without any arguments takes the user to
their home directory.

Examining Directory Contents : ls

Examining Directory Contents : ls

The contents of a directory can be listed using:
ls [-aAcCdfFgilLqrRstu1] filename
trinity% pwd
/report
trinity% ls
format
note

style

text

trinity% ls -l
-rwx------ 1 bill
-rw------- 1 bill
drwx------ 2 bill
drwx------ 2 bill

trinity% ls -F
format* note

style/

text/

trinity% ls /
mbox News report

trinity% ls -aF
./
format*
../
note

style/
text/

trinity% ls ..
mbox News report
trinity% ls /report/style
book letter

100
873
512
512

Mar
Mar
Mar
Mar

21
21
21
21

20:01
17:59
17:58
17:58

format
note
style
text

trinity% ls style
book letter
trinity% ls -d style
style
trinity% ls -ld style
drwx------ 2 greg 512 Mar 21 17:58 style

The ls command is used to display the contents of the specified directory. It takes
a variety of options that affect which files are displayed and the way their
information is formatted.
-l
-a
-g
-d
-F

long listing
all files, including those beginning with .
used with -l for group ownership
the directory file not its contents
show file type

Shell Wildcards

Shell Wildcards
Shell wildcards used for file name expansion
trinity% ls p*
pub pint plastered
trinity% ls /etc/*/m*
/etc/adm/messages
/etc/dp/modem
/etc/openwin/modules

Note that the shell expands wildcards

trinity% echo hello world
hello world
trinity% echo p*
pub pint plastered

The Unix shell provides wildcard expansion to generate filenames for commands.
To list all filenames that being with p (as above), then * is used to tell the shell to
generate the filenames automatically. The shell searches the specified directory to
find the files.
*
matches zero or more characters
?
matches exactly one character
[ABC] matches either A or B or C
[A-Za-z]
matches any single letter in upper or lower case
Note that wildcard expansion is different in Unix than for DOS. In DOS each
individual utility interprets * and may associate a different meaning to the symbol.
This is not possible in Unix, since the shell interprets the *, generates an argument
list, and then calls the specified command. In the above, echo simple writes to the
display its list of arguments; the list of arguments beginning with p were generated
by the shell prior to invoking echo.

Building the File System : mkdir/rmdir

Building the File System :

mkdir/rmdir
mkdir creates directories
rmdir removes them

trinity% rmdir biblio

trinity% ls -F
format* note
style/

trinity% pwd
/report
trinity% ls -F
format* note

style/

text/

text/

trinity% rmdir style

rmdir: style: Directory not empty

trinity% mkdir biblio

trinity% ls -F
biblio/ note
text/
format* style/

mkdir [-p] dir1 [dir2 ...]

rmdir [-ps] dir1 [dir2 ...]

The mkdir command is used to create new directories, and rmdir to remove
directories. In keeping with most Unix commands, the commands may be supplied
as many filename arguments as is required. In the following
mkdir one two three four five six /tmp/seven
six directories are created within the current directory, and a seventh is created
beneath /tmp. Notice, however, that a minimum of one directory must be supplied
to the command.
Using the -p option, mkdir is able to create missing parent directories as needed
mkdir -p first/second/third
will create the missing parent directories first and second if they do not already
exist.
Note that it is not possible to remove a directory with rmdir if it contains other files.
To remove the directory, first remove all the files and sub-directories which it
contains. The powerful (and somewhat dangerous) rm -r command is useful for
this.

Copying Files : cp

Copying Files : cp
cp copies files and directories around the
filesystem
trinity% ls -F
mbox
News/

report/

trinity% ls -F
mbox
News/

report/

trinity% ls -F report/style
book
letter

trinity% cp -r report/style .
trinity% ls -F
mbox
News/
report/ style/

trinity% cp report/style/book .
trinity% ls -F
mbox
News/
report/ book

trinity% ls -F style
book
letter

cp [-ip] f1 f2
cp [-ip] f1 f2 ... fn d
cp -r [-ip] d1 d2

cp is used to copy files and directories around the filesystem. Note that copy means
duplicating the bytes on disk representing the contents of the files being copied.
cp is used with two arguments when copying from one file to another and with
many arguments when copying a collection of files into a directory. In the case of
the latter, the directory must exist and be the last argument. cp may also be used to
copy the contents of one directory to another. In this case the -r (recursive) option
must be supplied. When copying directories, if the target (d2) exists, then the
source (d1) is created within it. A file f1 within d1 may now also be accessed as
d2/d1/f1. If, however, the target does not exist, then it is created and the actual
contents of d1 are copied into it. Therefore, a file f1 within d1, may now also be
accessed as d2/f1.
By default, the copy command overwrites any files which already exist with the
target name. The -i (interactive) option may be used in order to get cp to prompt
prior to overwriting any existing files.
To preserve a files modification time and permission bits, use the -p option. If it is
also necessary to preserve the files ownership, then the cpio command should be
used.

Copying with Wildcards

Copying with Wildcards

Wildcards may be used when copying multiple
files
trinity% cp *.c /home/george/src

Wildcards cannot be used for names which dont

exist
trinity% ls
chapter1.txt
chapter2.txt
trinity% cp *.txt *.bak

In DOS, the copy command does the expansion

copy *.c *.bak

Wildcards may be used with the cp command to generate a list of filename

arguments to be copied. However, it is important to realise that the cp command is
not involved in the expansion of the wildcard symbols. The wildcards are replaced
by the shell with any matching filenames, prior to invoking the command.
Since the shell generates the filename lists, based upon the files which currently
exist in the directory, it is impossible for the shell to generate names which do not
already exist. Therefore, the semantics of the DOS copy command do not apply to
the Unix cp command.
In the example given above, the user attempts to backup two chapters of a book, by
copying them to new names with the extension .bak. In DOS this would be
successful, in the older Unix systems it would be catastrophic. The shell would
replace *.txt with the two files and *.bak with nothing. It would then invoke the cp
command as follows
cp chapter1.txt chapter2.txt
which is most unfortunate since the purpose of the activity was to backup not
destroy the files!
Unix is not wrong in its behaviour, it is in fact entirely consistent. The semantics of
the DOS copy command are, however, somewhat strange. In DOS, the * in the first
argument is used as a wildcard and the * in the second as a place holder indicating
where to substitute the first part of the filename, and what to tack on the end.
Fortunately, recent shells pass the * to the application if they are unable to perform
an expansion and this gives rise to a cp error. The problem would be solved in
Unix by copying the files to a backup sub-directory.

Moving Files : mv

Moving Files : mv
Files and sub-directories can be moved
trinity% ls -F
mbox
News/

report/

trinity% ls -F report/style
book letter
trinity% mv report/style/book .
trinity% ls -F
mbox
News/
report/ book

trinty% ls -F
mbox
News/

report/

trinity% mv report/style .
trinity% ls -F
mbox
News/
report/ style/
trinity% ls -F style
book letter

trinity% ls -F report/style
letter

mv [-i] f1 f2
mv [-i] f1 f2 ... fn d
mv [-i] d1 d2

mv is used to rename files and directories. It does not cause the contents of the file
to be physically moved, only the files name is changed in its directory.
The new name may be a path to another directory, so mv can in fact move a file
from one place to another.
Note that there is no need for a recursive option when moving a directory since files
contained within the directory dont care what it is called. More specifically, the
contents of a directory file are the files stored within it, and mv does not effect file
contents. The -i option may be used if there is a danger of overwriting existing
files.

Deleting Files : rm

Deleting Files : rm
rm deletes files and directory structures
trinity% ls -F
mbox
News/
report/ book
trinity% rm book
trinity% ls -F
mbox
News/
report/

trinity% ls -F
mbox
News/
report/ style/
trinity% rm -r style
trinity% ls -F
mbox
News/
report/

rm -i requests confirmation before deletion

trinity% rm -i book
rm: remove book? n

trinity% rm -i book
rm: remove book? y

trinity% ls -F
mbox
News/

trinity% ls -F
mbox
News/

report/ book

report/

The rm command deletes files and directories. Beware that in Unix a deleted file is
lost forever. There is no mechanism to allow a file to be un-deleted since the disk
space associated with the file may immediately be re-used by some other process.
To recover a deleted file, the administrator must be asked to restore it from a system
backup. It is unlikely that the restored file will contain recent changes made to the
file.
The rm command also has the -i and -r options provided with cp. -i allows
interactive use of the command so that a use may stop the command from
accidently deleting a file. The -r option is necessary if directory structures must be
deleted.

Linking Files

Linking Files
Unix directories have multiple names
/

News

All directories have at least

two names, their name in
the parent directory and . in
themselves.

report

style

..

With each sub-directory, a

new name is created for the
parent, ..

The Unix file system is held together through links. Each file is identified by a link
name, or file name as it is usually called. Every file in the file system must have a
name (a link), however it is possible (and often necessary) that some files have
multiple links.
The above example shows the multiple names associated with directories. All
directories have at least two names, their name in the parent and . in themselves.
Should they have sub-directories, then a new name is generated for them in each
It is also possible to create multiple names for
sub-directory, .. The . and .. directory links are created automatically when a
files
new sub-directory
is made. They are used as a short-hand notation to refer to the
current or parent directories.

Linking Files : ln

Linking Files : ln
Use the ln command to create links
ln [-s] f1 f2

trinity% echo hello > afile

trinity% ls -l afile
-rw-r--r-- 1 fred
trinity% ln afile newName
trinity% ls -li afile newName
384927 -rw-r--r-- 2 fred
384927 -rw-r--r-- 2 fred
trinity% cat newName
hello

6 Jan 14 23:16 afile

6 Jan 14 23:16 afile

6 Jan 14 23:16 newName

The ln command enables users to create their own links.

In the above example, the names newFile and afile are linked to the same file. The
fact that afile existed first is not relevant, both are equal. Changes made to the file
through the name afile would be the same if made through the file name newFile.
The two names refer to exactly the same area on disk.

Hard links : ln

Hard links : ln
$ ln original new
By default, ln creates hard links
A hard link is simply another directory entry pointing
at the inode of the file or directory
- Hard links cannot be made across different partitions
or physical disks
- It does not matter if the original file or directory is
deleted, as this simply removes one of the hard links to
the file

The number of hard links to a file is recorded in the inode of the file, and can be
seen with ls.
trinity$ ls -li /usr/bin/vi
401851 -r-xr-xr-x
5 root
/usr/bin/vi

bin

227828 Jun 19

2002

The number of hard links to /usr/bin/vi is 5.

ex is one of the commands that is a hard link to the vi command
trinity$ ls -li /usr/bin/ex
401851 -r-xr-xr-x 5 root

bin

227828 Jun 19 2002 /usr/bin/ex

You can see that the inode number of the two files is the same.
When the number of hard links to a file or directory is 0, then there are no longer
any directory entries for the file, so it ceases to be available, i.e. it has been deleted.

Symbolic links : ln

Symbolic links : ln
$ ln s original new
With the -s option, ln creates symbolic links
A symbolic link is a file that contains as its data, the
absolute path to another file
- Symbolic links, unlike hard links, can cross file system
boundaries
- The link will break if the file or directory to which it
points is removed

In general, it is preferable to use symbolic links when creating links. This is

partially because they can cross over different partitions and physical disks, but also
because they are easier to see in listings.
Symbolic links can be seen using the ls command. Symbolic links will show their
file name and then the location to which they are linked.
e.g.

trinity$ ls -li bin

4 lrwxrwxrwx
1 root
bin -> ./usr/bin
trinity$ ls -dli /usr/bin
401792 drwxr-xr-x
4 root
/usr/bin

root
bin

9 May 13 15:06
9728 May 13 15:38

Note that since links are files in themselves, they will have their own inode number,
separate from that of the thing they point at.

Why create links?

Why create links?

Commands can be given different names to
indicate different behaviour
The vi editor has multiple names to provide different
functionality depending on how it is started

Files may be linked to appear in multiple

directories
A file can be linked to appear in several places in the
file hierarchy and can be edited from any of them,
however, the data appears the same in all places.

Backwards compatibility

Allows users to use an old name for a renamed file

It is sometimes useful for files to have multiple names.

For example, the standard Unix editor, vi, has several names. The different names
for vi include ed, view and vipw. The purpose of the multiple names is to give the
illusion that there are many programs providing vi like editing facilities rather than
simply one.
When invoked, the vi program checks which name was used to invoke it and
changes its behaviour accordingly. This saves users having to remember large
number of command line options
A useful application of links is to give the illusion that a file exists in multiple
directories. This is achieved by specifying path names with the ln command. In the
event that the directories physically exist on different partitions, the -s option (for
symbolic) must be used to establish the link.

Pathname abbreviations

Pathname abbreviations
$HOME and ~
The home or login directory of the user

~username
The home or login directory of the user username

$PWD and .
The present working directory

..
The parent directory

~ is only available in ksh, csh and bash

Shell Meta-characters

Shell Meta-characters
Characters with special meanings to the shell
e.g.
* ? [ ] ` < > | ! ~ $ %
The dot ., underscore _ and hyphen - are
not meta-characters
It is advisable not to use these meta-characters
in directory or file names

Meta-characters have special meanings to the shell.

However, as there are a limited number of characters available, they may also have
special meanings to other programs.
You will need to be aware of what is interpreting the character to be sure of what
meaning is assigned to it.

* - Asterisk

* - Asterisk
* represents zero or more of any character
trinity% ls
dante dir dir2 dir3 eat fruit fruit2 zilog
trinity% ls d*
dante dir
dir2:
beans coffee nuts
dir3:
mango peach pear
trinity%

?- Question Mark

?- Question Mark
? represents any single character
trinity% ls
dante dir dir2 dir3 eat fruit fruit2 zilog
trinity% ls dir?
dir2:
beans coffee nuts
dir3:
mango peach pear
trinity%

[ ] Square Brackets

[ ] Square Brackets
[] defines an occurrence of a range
trinity% ls
dante dir dir2 dir3 eat fruit fruit2 zilog
trinity% ls [e-z]*
eat fruit fruit2 zilog
trinity% ls dir[0-5]
dir2:
beans coffee nuts
dir3:
mango peach pear
trinity%

Ranges are matched using ASCII

Each character is translated to the equivalent ASCII value and then the match is
performed mathematically.
Using decimal ACSII values, the range [e-z] would be translated as any character
with an ASCII value between 101 and 122, i.e. 101 < character value < 122
It is not possible to specify a range of [z-e] as this would translate to a
mathematical match of less than 122 and greater than 101 which is not possible to
represent in one mathematical expression.

` ` - Backticks

` ` - Backticks
` ` - backticks (or graves) force the command
enclosed in them to be executed and its output
substituted into command before the remainder
of the line is evaluated
trinity$ rm `cat obsolete_files.txt`
trinity$ YEAR=`date +%Y`
trinity$ echo $YEAR
2005
trinity$ echo the hostname is uname -n
the host name is trinity

Backticks are particularly useful in shell programming.

Quoting in Shells [ ]

Quoting in Shells [ ]
Use single quotation marks to identify text
literally
trinity$ echo the path is $PATH
the path is $PATH

Use double quotation marks to identify text

literally, while enabling variable and command
expansion
trinity$ echo the path is $PATH
the path is /bin:/sbin:/usr/ucbbin

Why quote?
We have seen that the shell has meta-characters which have special meanings. We
have also seen that some of these characters have meanings to commands. We can
use quoting to force meta-characters to be dealt with in the way we expect, and by
the command we expect.

Escaping in shells [ \ ]

Escaping in shells [ \ ]
\ - The back-slash escapes the the meaning of
the following character so that it the shell does
not interpret it, if it normally would
trintiy$ echo \the path is \$PATH\
the path is $PATH

The backslash - \ - allows single characters to be escaped so that they are not
interpreted by the shell. This can be particularly useful for escaping quotes or
single characters that would still be interpreted inside quotes.

Getting Help

Getting Help
Good information is hard
to get. Doing anything with
it is even harder !

L. Skywalker

Getting help : man

Getting help : man

man displays sections of the Unix manual
You must know the name of the command you
want help with
trinity% man head
trinity%

Displays the manual pages on the command

head

The standard way for getting help in Unix is the man command.

Getting help : man

If you do not know quite what the command is
trinity$ man k print

provides a list of command which are related to print

apropos does the same

trinity$ apropos print

Both assume the System Administrator has built

whatis database
trinity$ catman -w

The standard way of using man requires that you know the name of what you want
help about.

Getting help : man

To move around the man page, use the
following commands

Spacebar
b
f
q
h
/string
n

moves you a single screen full

moves back a single screen
moves forward a single screen
leave the manual page display
lists all the scrolling facilities
searches forward for string
finds the next occurrence of string

man : Output

man : Output
Header line
User Commands

head(1)

The section of the manual the command belongs to

Name
head - display first few lines of files

The name of the command and and an outline

Synopsis
head [ -number | -n number ] [ filename...

The command and its options and parameters

Description
A brief description of what the command does

man : Output
Options
A detail explanation of each of the options
supported by the command

Operands
A list of parameters, such as files, this command can
manipulate

Usage
How the command is used and what problems are
know about it

Examples
A simple example

man : Output
Environment
Any environmental variables which the application can
or does use.

Exit Status
What error levels or exit status are given when the
program has run
[very useful when writing scripts]

Attributes
Attributes of the command

See Also
commands associated with this one

man : example

man : example
User Commands

head(1)

NAME
head - display first few lines of files
SYNOPSIS
head [-number | -n number]

[filename...]

DESCRIPTION
The head utility copies the first number of lines of each
filename to the standard output. If no filename is given,
head copies lines from the standard input. The default value
of number is 10 lines.
When more than one file is specified, the start of each file
will look like:
==> filename <==
Thus, a common way to display a set of short files,
fying each one, is:
example% head -9999 filename1 filename2 ...

identi-

man : example
OPTIONS
The following options are supported:
-n number
The first number lines of each input file will be
copied to standard output. The number option-argument
must be a positive decimal integer.
-number
The number argument is a positive decimal integer with
the same effect as the -n number option.
If no options are specified, head will act as
been specified.

if

-n

10had

OPERANDS
The following operand is supported:
file A path name of an input file. If no file operands
specified, the standard input will be used.

are

man : example
USAGE
See largefile(5) for the description of the behavior of head
when encountering files greater than or equal to 2 Gbyte ( 2
**31 bytes).
EXAMPLES
Example 1: Writing the first ten lines of all files
To write the first ten lines of all files (except those with
a leading period) in the directory:
example% head *
ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment
variables that affect the execution of head: LANG, LC_ALL,
variables that affect the execution of head: LANG, LC_ALL,
LC_CTYPE, LC_MESSAGES, and NLSPATH.

man : example
EXIT STATUS
The following exit values are returned:
0

Successful completion.

>0

An error occurred.

ATTRIBUTES
See attributes(5) for descriptions of the
butes:

following

attri-

____________________________________________________________
|
ATTRIBUTE TYPE
|
ATTRIBUTE VALUE
|
|_____________________________|_____________________________|
| Availability
| SUNWcsu
|
|_____________________________|_____________________________|
| CSI
| enabled
|
|_____________________________|_____________________________|
| Interface Stability
| Standard
|
|_____________________________|_____________________________|
SEE ALSO
cat(1), more(1), pg(1), tail(1), attributes(5),
largefile(5), standards(5)

environ(5),

man page sections

man page sections

1

User Commands

System Calls

C library functions

File Formats

Standards, Environments and Macros

Games and Demos

Device and Network Interfaces

System Administration

Driver Entry points

Some sections have modifiers which group similar pages.

e.g.
1M
System maintenance and administration commands
3LIB
Libraries implemented a shared object

manual entries in other sections

manual entries in other sections

The SEE ALSO part at the bottom of a man page
lists alternate references.
If the command is followed by a number other
than 1 in parentheses, it indicates a section of
the man pages
passwd(4)

To view this section, use

trinity$ man s 4 passwd

This will display the format of the password file

Other useful man options

Other useful man options

Most sections of the manual have an
introduction
man -s# Intro

If a keyword appears in multiple sections of the

manual, these can be listed
man -l keyword
trinity$ man -l passwd
passwd (1)
-M /usr/share/man
passwd (4)
-M /usr/share/man

Introductory pages explain what the section of the manual contains, any subsections
and often list the pages within the section.

Google

Google
Help can often be found on-line through search engines
such as Google
http://www.google.com/
Searches can be restricted using limiters
site
filetype

Logical constructs can be used to improve results

+

forces the inclusion of common words

forces the exclusion of a word

OR

either one search term OR a different search term

Google: Example

Google: Example

SunSolve

SunSolve
SunSolve is Sun's official online help source
http://sunsolve.sun.com/
Available to the general Sun Community:
-

Security Information
Resolved Sun Alerts
Patch Descriptions (a.k.a. Patch ReadMe documents)
Archived SunSolve content
Limited Access to the SunSolve Knowledgebase
Sun Support Forums
Big Admin

Extras available to Service Plan or Contract customers:

- Full Access to the SunSolve Knowledgebase

File Types and Listing

File Types and Listing

The human animal differs from

the lesser primates in his
passion for lists of Ten Best.
H. Allen Smith

Varieties of Files

Varieties of Files
Ordinary Files
ASCII text or binary data, formatting left to user
Subclass of ordinary is hidden (name starts with .)

Directory
Directory provides a connection between the names
of the files and the files themselves. They impose a
structure on the file system

Device Files
Often located in the /dev directory
Writing to the device file transfers data to the device.
A directory is a file whose data is a list of file names and the locations of the inodes
that describe those files.
Device files are used to access hardware. They come in two types, block and
character.

What sort of file [file]

What sort of file [file]

$ file filename
classifies the named files
several tests performed on file
- if plain text, tries to determine programming language
used
- if binary, compares the magic number, if available,
with contents of /etc/magic to determine type
trinity$ file dante
dante: English text
trinity$ file /usr/bin/cat
/usr/bin/cat: ELF 32-bit MSB executable SPARC
Version 1, dynamically linked,stripped

The file command is used to provide information about the contents of the file
passed as the parameter.
It performs a number of tests on the data in the file, including whether the file starts
with a particular magic number.
The file /etc/magic contains a plain text listing of these magic numbers and what
they mean about the file.

Strings in a file [strings]

Strings in a file [strings]

$ strings filename
Show any ASCII strings within a file

trinity$ strings /usr/bin/cat

SUNW_OST_OSCMD
usvtebn
usage: cat [ -usvtebn ] [-|file] ...
cat: Cannot stat stdout
cat: cannot open %s
cat: cannot stat %s
cat: input/output files `%s' identical
cat: close error

The strings command can be used to find ASCII strings within a binary file.
This can be useful for finding error messages in binary files, or identifying random
binaries.

Contents of a file [cat]

Contents of a file [cat]

$ cat filename
Displays the contents of the files to stdout, usually, the screen
trinity$ cat dante
Half way along the road we have to go,
I found myself obscured in a great forest,
Bewildered, and I knew I had lost the way.
It is hard to say just what the forest was like,
How wild and rough it was, how overpowering;
Even to remember it makes me afraid.
...

The cat command displays the contents of a file to the standard output file handle.
This is usually the screen.
If given more than one file name, cat will display the files sequentially. This can be
used to join, or concatenate, files.

Listing contents of files [more]

Listing contents of files [more]

$ more filename
Display the contents of a file a small quantity at a time
to stdout
<cr>
f
space
q
h
/string
n

moves you forward a single line

moves you forward a single screen
moves you forward a single screen full
leave the page display immediately
lists all the scrolling facilities
searches forward for string
finds the next occurrence of string

The more command is one of a group of commands known as pagers, i.e. they
display data a page at a time.
An alternative to more is the less command.

Listing contents of files [less]

Listing contents of files [less]

$ less filename
Display the contents of a file a small quantity at a
time to stdout
less is the opposite of more
Allows backwards movement through file
Doesnt read whole of file before displaying
- Useful for large files

NAME
less - opposite of more
SYNOPSIS
less -?
less --help
less -V
less --version
less [-[+]aBcCdeEfgGiImMnNqQrsSuUVwX]
[-b bufs] [-h lines] [-j line] [-k keyfile]
[-{oO} logfile] [-p pattern] [-P prompt] [-t tag]
[-T tagsfile] [-x tab] [-y lines] [-[z] lines]
[+[+]cmd] [--] [filename]...
(See the OPTIONS section for alternate option syntax with
long option names.)
DESCRIPTION
Less is a program similar to more (1), but which allows
backward movement in the file as well as forward movement.
Also, less does not have to read the entire input file
before starting, so with large input files it starts up faster than text editors like vi (1).

Top few lines [head]

Top few lines [head]

$ head [-#] filename
Display the first 10 lines of a file by default to stdout
[-#] change the number of lines displayed

trinity$ head 3 dante

Half way along the road we have to go,
I found myself obscured in a great forest,
Bewildered, and I knew I had lost the way.

NAME
head - display first few lines of files
SYNOPSIS
head [-number | -n number] [filename...]
DESCRIPTION
The head utility copies the first number of lines of each
filename to the standard output. If no filename is given,
head copies lines from the standard input. The default value
of number is 10 lines.

Bottom few lines [tail]

Bottom few lines [tail]

$ tail [-#] [-f] filename
Display the last 10 lines of filename by default to
stdout
[-#] change the number of lines
[-f] forever loop displaying end of file

trinity$ tail 3 dante

That are in the heavens, through a round opening;
And then we emerged to see the stars again

NAME
tail - deliver the last part of a file
DESCRIPTION
The tail utility copies the named file to the standard output beginning at a designated place. If no file is named,
the standard input is used.

Fields in a file [cut]

Fields in a file [cut]

$ cut -b list [-n] [file...]
$ cut -c list [file...]
$ cut -f list [-d delim] [-s] [file...]
-b

list refers to bytes

-c

list refers to characters

-f

list refers to fields

list

comma or space separated list of numbers

indicating the required items

[-n]

do not split multi- byte characters

[-d delim]

field delimiter character, default is tab

[-s]

suppress lines with no delimiter characters

NAME
cut - cut out selected fields of each line of a file
SYNOPSIS
cut -b list [-n] [file...]
cut -c list [file...]
cut -f list [-d delim] [-s] [file...]
DESCRIPTION
Use the cut utility to cut out columns from a table or
fields from each line of a file; in data base parlance, it
implements the projection of a relation. The fields as
specified by list can be fixed length, that is, character
positions as on a punched card (-c option) or the length can
vary from line to line and be marked with a field delimiter
character like <TAB> (-f option). cut can be used as a
filter.

Counting things [wc]

Counting things [wc]

$ wc [-c] [-l] [-w] filename

Counts contents of a file

[-c] characters
[-l] lines
[-w] words.

trinity$ wc dante
33 223 1320 dante
trinity$ wc -l dante
33

NAME
wc - display a count of lines, words and characters in a
file
SYNOPSIS
wc [-c | -m | -C] [-lw] [file...]
DESCRIPTION
The wc utility reads one or more input files and, by
default, writes the number of newline characters, words and
bytes contained in each input file to the standard output.
The utility also writes a total count for all named files,
if more than one input file is specified.
wc considers a word to be a non-zero-length string of characters delimited by white space (for example, SPACE, TAB).
See iswspace(3C) or isspace(3C).

Finding the differences [diff]

Finding the differences [diff]

$ diff [-i] [-c] file1 files2
Compares contents of two files
[-i] case insensitive comparison
[-c] position contextual differences

trinity$ diff /etc/hosts hosts

20c20
< 192.168.1.2
tiny
-->

NAME
diff - compare two files
SYNOPSIS
diff [-bitw] [-c | -e | -f | -h | -n | -u] file1 file2
diff [-bitw] [-C number | -U number] file1 file2
diff [-bitw] [-D string] file1 file2
diff [-bitw] [-c | -e | -f | -h | -n | -u] [-l] [-r] [-s]
[-S name] directory1 directory2
DESCRIPTION
The diff utility will compare the contents of file1 and
file2 and write to standard output a list of changes necessary to convert file1 into file2. This list should be
minimal. Except in rare circumstances, diff finds a smallest
sufficient set of file differences. No output will be produced if the files are identical.

Differences of 3 files [diff3]

Differences of 3 files [diff3]

$ diff3 file1 file2 file3
Displays the differences between 3 files
trinity$ diff3 /etc/hosts /etc/inet/hosts hosts
====
1:20c
192.168.1.2
tiny
2:20c
3:20c
192.1.6.1

unplugged

NAME
diff3 - 3-way differential file comparison
SYNOPSIS
diff3 [-exEX3] filename1 filename2 filename3
DESCRIPTION
diff3 compares three versions of a file, and publishes
disagreeing ranges of text

Sorting files and output [sort]

Sorting files and output [sort]

$ sort [OPTIONS] files
Ordering options
- -d
- -f

dictionary order
fold lower case into upper case
i.e. a follows A
- -n
numerically
- -M
compares months
i.e. Jan before Feb before Mar
- -r
reverses order
Use +n to sort on a particular column, where n is the column number
multiple columns may be sorted at the same time
- ordering options can be specified for each column

sort +6n +3r file

sorts the file numerically by the sixth column and in reverse order by
the third column

Sort can be used to sort multiple files into one sorted file.
Sort is often used as part of pipelines to sort the output of one command before it is
input into another.

Handling duplicate lines [uniq]

Handling duplicate lines [uniq]

$ uniq [input_file] [output_file]
Reports or filters repeated lines in a file
Only works if lines are adjacent
- Run sort first on input
trinity$ cat file
2
6
4
6
5
trinity$ sort -n file | uniq
2
4
5
6

NAME
uniq - report or filter out repeated lines in a file
SYNOPSIS
uniq [-c | -d | -u] [-f fields] [-s char] [ input_file
[output_file]]
uniq [-c | -d | -u] [-n] [ + m] [ input_file
[output_file]]
DESCRIPTION
The uniq utility will read an input file comparing adjacent
lines, and write one copy of each input line on the output.
The second and succeeding copies of repeated adjacent input
lines will not be written.
Repeated lines in the input will not be detected if they are
not adjacent.

Redirection

Redirection

The trail's got to be 'round here

somewhere!

D. Boone

Why ?

Why ?
For useful work to be performed data must be
transformed
Most work requires data to be transformed in a
number of different ways

Unix commands designed to do one task very

well
Redirection allows commands to be joined
together in an efficient way

Redirection is an important part of the Unix philosophy.

Commands are designed to do one job extremely well. However, most real tasks
require more than one transformation to be performed on the input data in order to
get out the required information.

Standard Files

Standard Files
stdin

aka file descriptor 0

This file normally be referenced by scanf();

Typically the keyboard

stdout

aka file descriptor 1

This file normally referenced by printf();

Typically the display

stderr

aka file descriptor 2

This file to which run time errors are sent;

Typically the display

Every process is assigned three files on creation; stdin, stdout and stderr.
These files are used to accept input, display output and display errors.

Redirection : < > 2>

Redirection : < > 2>

stdin, stdout, and stderr may be redirected
use file descriptors and the < and > symbols
$ command < file
Run the command but get input from the file file instead of
the keyboard
$ command > file.out
Run the command but put the output in the file file.out
instead of on the display
$ command 2> errors.lst
Run the command and put any errors produced from the
command into errors.lst

is redirected using the less than symbol - <

This allows commands to receive input from files, rather than the keyboard
stdin

is redirected using the greater than symbol - >

This allows the output produced by commands to be sent to a file, rather than the
display
stdout

is redirected using its file descriptor 2 and the greater than symbol - 2>
This allows any errors output by the running command to be sent to a file, rather
than the display.
This can be particularly useful in scripts.
stderr

Redirection : >>

Redirection : >>
>
Causes a new file to be created each time the redirection occurs

>>
Causes the text to be appended to an existing file or creates a
new file if one does not exist

trinity$ echo Smith:Bubba:Seymour >names

Creates a new file or overwrites file called names

trinity$ echo Smith:Bubba:Seymour >>names

Adds Smith:Bubba:Seymour to the end of the file names

Single redirection arrows are used to create new files. If the file already exists, it is
overwritten by the new output.
Double redirection arrows are used to append data to files. If the file does not exist,
it will be created. However, if the file already exists, the output will be added to the
end of the file.

Redirection : Merging

Redirection : Merging
Sometimes useful to merge stderr and
stdout
for scripts, where we want to capture the output and
any errors in the same file

Merging is simply a redirection of stderr to

stdout
stdout is referenced by its file descriptor (i.e. 1)
2>&1

Redirection: Multiple redirections

Redirection: Multiple redirections

Multiple redirections can be used with a single command
trinity$ command < monthly_stats > report 2>&1

Error output merged with standard output

Standard output to file called report
Input from file called monthly_stats

Order is important with multiple redirections

Read the redirections from right to left
cp > out.a 2>&1

cp 2>&1 > out.b

Only out.a will contain the usage message from the cp

command

A very powerful way of using redirection, is to use more than one redirection with a
single command.
It is important to note that redirections are performed in a particular order, and
should be read from right to left.
Thus, in the example shown:
cp > out.a 2>&1

This reads as: redirect stderr to stdout (i.e. merge stderr and stdout) then
redirect stdout to the file out.a
cp 2>&1 > out.b

This reads as: redirect stdout to out.b, then redirect stderr to stdout.
However, as stdout has already been redirected, it is not possible to redirect
stderr to stdout, so the usage message from cp will go to the display.

Redirection : Examples

Redirection : Examples
trinity$ cat dante > newdante

Takes the output from cat, i.e. the contents of the

file dante, and places it in a new file newdante
- Effectively, this is a replacement for the cp
command
trinity$ echo bin:x:2:2:bin:/bin:/bin/sh >> /etc/passwd

Will append onto the end of the /etc/passwd file a

new username bin
- This is a quick way to add new users to the
system

More redirection: File creation

More redirection: File creation

trinity$ echo Hello World > myworld

Creates a file myworld and stores Hello World

trinity$ cat > myfile
And if anyone knows anything about anything,
said Bear to himself, its Owl who knows some
-thing about something, he said, or my names
not Winnie-the-Pooh, he said. Which it is, he
added. So there you are.
Ctrl-D

Creates a file myfile by entering text via the

keyboard until the Ctrl-D is entered when the file is
closed and saved

Another useful method of file creation using redirection is called the Here
Document.
This is a specialised case of the second example shown, where text read from the
following lines until an end of file marker (typically the string EOF, but may be
anything) is entered.
cat > newfile <<EOF
And if anyone knows anything about anything,
said Bear to himself, its Owl who knows some
-thing about something, he said, or my names
not Winnie-the-Pooh, he said. Which it is, he
added. So there you are.
EOF

As in the example in the slide, a file called new file would be created containing the
data, however, this method is useful for scripts where one may wish to create long
files, e.g. containing configuration data, without using multiple echo statements

Pipelines : |

Pipelines : |
Pipes are command line FIFO buffers

redirects the stdout of one file to the stdin of

another
the pipe is used to supply input to another command
redirection will direct output to other than a
command
$ command1 | command 2

The output of command1 is used as the input

of command2

Pipelines : examples

Pipelines : examples
trinity$ ls . | grep morpheus

Lists the current directory and searches for the string

morpheus
- find files named morpheus in current directory

trinity$ host -t MX matrix.net | grep 5 | cut -d" " -f7

Lists the mail servers belonging to the domain

matrix.net, searches for the number 5 and
returns the seventh field from any matching line,
using spaces as the field delimiter
It can be seen that very complex transformations can be built using pipelines.

Named Pipes

Named Pipes
Named pipes, or FIFOs, are used to facilitate
communications between a sending process and
a receiving process.
$ mknod name p
$ mkfifo [-m mode] path
Both commands will create a named pipe, either
called name or located at path

A FIFO is a queue where the first item in, is the first item out.
Named pipes can be used instead of temporary files in scripts where it is necessary
to pass data between processes.

File Security

File Security
Anyone who uses the phrase
as easy as taking candy from
a baby, has never tried taking
candy from a baby
R. Hood

Security Overview

Security Overview
The primary function of a system's security
feature is to deny access to unauthorized users
Unixs primary security features
User passwords
File and directory protection with permissions
Files that control remote logins and commands
on individual workstations
Logs and audit files for System Administrators
to check for unauthorized usage

The requirement for security comes directly from the multi-user, networked nature
of Unix.
By requiring users to log into the system, we can restrict access to systems to only
those people who need to access them. It also helps provide an audit trail as to who
has done what on a system. This is an important tool for administrators and is a
very good reason why you shouldnt allow other people to use your account.
Since the system has a concept of identity, security can be further refined, by
assigning ownership to files, then defining how this affects a users ability to read,
write or execute the file.

Permissions

Permissions
Unix has two default levels of security
Username and passwords to access the workstations
File access permissions

Files and directories are automatically protected

by default system permissions when they are
created

Usernames and passwords limit access to a system and provide identity once logged
in.
File and directory permissions provide access control within the file hierarchy,
limiting what users can see and do on a system.

Finding the permissions [ls l]

Finding the permissions [ls l]

trinity$ ls l thisfile
-rwxrw---x morpheus users

34342 Jan 14 1999

thisfile

File type - Kind of file

- d for Director y
- l for Symbolic Links
- - for ordinar y files
- p for named pipes
User (owner)
- The user who created the file or directory
Group
- Class of users defined by the system administrator
Others (public)
- All other users

The mode of the file details its type and the permissions of the file, as stored in the
inode. It is listed in the first 10 characters of the output of the ls command when
the -l parameter is used.
The first character may be one of the following:
d
The entry is a directory.
D
The entry is a door.
l
The entry is a symbolic link.
b
The entry is a block special file.
c
The entry is a character special file.
p
The entry is a FIFO (or "named pipe") special file.
s
The entry is an AF_UNIX address family socket.
The entry is an ordinary file.
The remain nine characters should be examined as three groups of three and show
the permissions granted to the user who owns the file, the group of users the file is
associated with, and any other user who does not fall into one or other of the
previous categories.

Types of users

Types of users
User
Owner of a file may manipulate a file in any manner, except
transferring ownership
Only the super-user may change the user who ow ns a file

Group
A list of users, as in a class, all of whom have equivalent
permissions
To find w ho is in a group, look in the /etc/group file

Others
Public with a login on the system may access the file

The super-user, root, is a special case, and ignores all permissions on all files. This
can be very dangerous.

Access Permissions

Access Permissions
Access permissions affect files and directories in
different ways
readable
Files:
Directory:

May be viewed internally by user with proper

status
Contents may be listed

writable
Files:
Directory:

May be overwritten by a user with proper status

May have new files added and removed

executable
Files:
Directory:

May be executed by user with proper status

Allows directory to be traversed by users or
processes

Directories are files whose data consists of a list of file names and pointers to the
inodes of those files.
If a user has read access on a directory, then they may list the contents of that
directory, i.e. the data of the file is visible.
If a user has write access on a directory, they they may create and delete files in the
directory, i.e. modify the data in the directory file, by adding or removing links
between file names and inodes.
If a user has execute permissions on a directory, they are allowed to enter the
directory or pass through it.

How is access decided ?

How is access decided ?

If the UID of the user is that of the Super-User,
ignore all permissions and grant full access
If the process UID and the file UID match
Use the User Permissions bits

If the process GID and the file GID match

Use the GID Permissions bits

If neither UIDs nor GIDs match

Use the Other permissions

For users other than root, access is processed using the most restrictive match first.

Controlling access [chmod]

Controlling access [chmod]

$ chmod [-R] mode file
Sets the access rights on file
[-R] recursive
mode contains those access permissions
- may be symbolic or octal
trinity$ chmod o=rwx newfile
trinity$ chmod R 755 newdir

As with rm, it can be dangerous to use chmod recursively.

Controlling access [chmod]: The modes

Controlling access [chmod]: The

modes
File access modes can be specified in two ways
Symbolic
Who
What

u user
r read

g group o other
w write x execute

How

= set

+ add

a all

- remove

Octal

- group of three numbers

- position specifies who permissions is defined for
- U## : user, #G# : group, ##O : others

- value specifies permission

- 4 : read, 2 : write, 1 : execute

- multiple permissions may be set by summing values

When using symbolic modes, a is used to set a permission for all of the user, group
and others. If who the permission is for is omitted, then a is assumed.
For example, to specify that a file may be read and modified by its owner, and the
group, but only read by anyone else use the following modes
Symbolic:
chmod ug=rw,o=r myfile or chmod +r,ug+w
Octal: chmod 664 myfile
It is often easier to set permissions initially with an octal mode and then make
changes using the symbolic notation.

Octal Modes

Octal Modes
user

group

other

read

400

040

004

write

200

020

002

execute

100

010

001

Octal Modes : example

Octal Modes : example

-rwx rw- --x

r
1

root

w
x
1
1
user
4+2+1=7

other

r
1

34342

w
x
1
0
group
4+2+0=6

Jan 14 1999

thisfile

r
0

w
x
0
1 =761
other
0+0+1=1

In this example, we can see that the file has been given the following permissions
User (owner): may read, modify and execute the file
Group:
may read and modify the file
Other:
may execute the file
The octal mode which matches this permission set is 761

User ID and Group ID

User ID and Group ID

Every user has a unique number that identifies files and
processes belonging to them
The number is know n as a UID (user ID)
UIDs are mapped to names in the /etc/passwd file

Every group is also identified by a unique number

The number is know n as a GID (group ID)
GIDs are mapped to names in the /etc/group file

A user's primary group often has the same name as the

user and the UID and GID are usually the same.

UIDs and GIDs can also be mapped to names using naming services, such as LDAP
or NIS. These are commonly used in large networked environments so that users
and groups are provided consistently across the entire network but can be managed
from a central location.

Real & Effective User/Group ID

Real & Effective User/Group ID

real user and group id from /etc/passwd
effective initialised as same but can be
changed
real user/group is who is actually running
effective is for determining permissions
Why would you want them to be different?

Non-privileged users accessing privileged information

We have seen that ordinary users do not have access to the shadow file where
passwords are stored.
How then can a user change their own password?

Who am I [id]

Who am I [id]
$id [user]
Display the UID, GIDs and corresponding user names
and group names for the current, or specified, user
If real and effective UID and GID are not the same,
both will be displayed

trinity$ id
uid=115(guest) gid=10(staff)

The id command is particularly useful for finding the effective UID and GID of a
user.

Ownership

Ownership
trinity$ ls an thisfile
-rwxrw---x 1002
100

34342

Jan 14 1999

thisfile

All files have a UID and GID associated with them

determining the ownership
trinity$ ps -ef | grep ssh
root
315
1 0 10:28:25 ?
root
327
315 0 10:28:48 ?

0:00 /usr/lib/ssh/sshd
0:01 /usr/lib/ssh/sshd

All Unix processes have a UID and GID associated

with them
trinity$ id
uid=1002(morpheus) gid=100(users)

UIDs and GIDs are mapped to names by the

/etc/passwd and /etc/group files or naming
services, such as LDAP or NIS
By default, processes are created with the UID and GID of the owner and group
owner of the executed file.

Changing Ownership: chown and chgrp

Changing Ownership: chown and

chgrp
$ chown [-fhR] owner [ : group] file
$ chgrp [-fhR] group file
[-f]
[-h]
[-R]
owner
group

forces changes, no errors reported

if the file is a link, change the ownership of the link,
not the referenced file
make changes recursively
UID or name of user to ow n file
GID of name of group to own file

Only the super-user may change the owner of a file

trinity$ ls l
drwxr-xr-x
2 root
-rw-r--r-1 root

other
other

117 Apr 19 13:04 newdir

0 Apr 19 13:04 newfile

trinity$ chown morpheus newfile ; chgrp users newdir

trinity$ ls l
drwxr-xr-x
2 root
users
117 Apr 19 13:09 newdir
-rw-r--r-1 morpheus other
0 Apr 19 13:09 newfile

chown can be used to change the ownership and group ownership of a file.
Generally, it is only used by the super user.
chgrp can be used to change the group ownership of a file.
As with rm, it can be dangerous to use chown or chgrp recursively.
Standard users may only have one real user id, but may be members of many
groups.
The super-user is not restricted by file permissions or ownership.

setuid

setuid
chmod +s filename
Program file owned by one user but run by another
Program runs as though by file owner
-r-sr-sr-x

1 root

sys

21964 Apr

2002 /usr/bin/passwd

root owns the file, so allow root to run program on your

behalf (e.g. change your password)
Routine in program can now make setuid call

passwd command runs as root, no matter which user invokes it. This allows it
access to the shadow file, which is not accessible by non-root users.
setuid should be used with care on root owned files as it removes some of the
protection granted by ownership and access permissions.

setgid

setgid
setgid operates in a similar fashion to setuid
When program is run it takes the permissions of
the program's group rather than the group(s) of
the user
chmod g+s filename

If a program has the has the set-group-ID bit set, then when any user runs the
program it will be run with all the access privileges of the program's group rather
than the group(s) of the user running the program. A program can give up this
privilege after starting (which is often done for security reasons).

setuid and setgid with directories

setuid and setgid with directories

Directories can also have the setuid or setgid
bits set
In this case, files created in the directory will

be owned by the user who owns the directory, or

be owned by the group the directory belongs to, or
both
rather than being given the UID or GID of the
creating user

If a directory has these bits set, everything created under that directory will be
owned by the directory's user or in the group of the directory, regardless of who
creates the file.

Sticky bit

Sticky bit
chmod

+t file

On a directory

Restricts who may rename or delete files in directory

Used on /tmp and /var/mail to prevent users
deleting files which belong to other users
ls -ld /tmp /var/mail
drwxrwxrwt
5 root
drwxrwxrwt
3 root

sys
mail

312 May 15 16:57 /tmp

512 May 13 15:06 /var/mail

On a file

Used to prevent swapping out of executable

No longer used on modern operating systems

ls displays a set sticky bit as a "t" in the permissions of a file or directory

If a directory is writable and has the sticky bit set, files within that directory can be
removed or renamed only if one or more of the following is true:
o the user owns the file
o the user owns the directory
o the file is writable by the user
o the user is a privileged user
/tmp is owned by root and
If set for an executable, the sticky bit tells the kernel to keep the code loaded in
swap space even after it has finished executing on the assumption that it is likely to
be used again soon. This performance optimisation was included in some early
versions of Unix to save reloading frequently used programs such as the shell or vi
from disk.

Default Permissions [umask]

Default Permissions [umask]

$ umask [mode]
Displays or sets the umask, default file creation value
[mode] contains a 3 byte octal value
trinity$ umask
022
trinity$ umask 027
trinity$ umask
027

The operating system sets a default umask

Users can override this umask in their .profile

The usual default umask is 022.

This means that files are created readable and writable by their owners and only
readable to the group and the rest of the world.
Owners of directories created with this umask may list the files in the directory and
create and delete new files. Users in the group and other users may only list the
files, change into the directory or traverse through it.

Default Permissions [umask]

Default object creation permissions are set by
the current umask
umask shows the permissions NOT granted on
the file or directory
For file creations, subtract the umask value from
666.
For directory creations, subtract the umask value
from 777
trinity$ umask
022
trinity$ touch newfile; mkdir newdir
trinity$ ls l
drwxr-xr-x
2 morpheus users
-rw-r--r-1 morpheus users

117 Apr 18 10:43 newdir

0 Apr 18 10:43 newfile

The maximum permissions that can be set on a file are all of the user, group and
other having read and write permissions. The octal mode which corresponds with
this permission set is 666.
Files are not created executable, by default, for security reasons. Most files will
never be executed.
The maximum permissions that can be set on a directory are all of the user, group
and other having read, write and execute permissions. The octal mode which
corresponds with this permission set is 777.
Directories are created executable, by default, so that they may be traversed.

Searching and Translating

Searching and Translating

Were you looking for me?

Dr. Livingstone

Why?

Why?
Useful work transforms data
This is often achieved using some form of
translation
It is useful to be able to find data
So that we can transform the right thing
So that we only transform what is necessary

find : Finding files

find : Finding files

$find path condition
Descends the file system tree beginning at path
locating files which matches the condition
specified in condition

$find . name fruit

Finds a file whose name is fruit in this directory
and any below and displays the name of that file

find : Conditions

find : Conditions
$ find path condition

[-user uid]
[-group gid]
[-mtime [+|-] n]
[-atime [+|-] n]
[-perm nnn]

[-inum n]
[-size [+|-] n]
[-exec command {}\;]

file whose owner is uid

file whose group is gid
file modified in the last n days
file accessed in the last n days
file with permissions matching
octal nnn
file with i-node number of n
file of size bigger or smaller
than n blocks
run command on each match
with the current match
replacing {} in the command to
run.

Block size is dependant on file system, however, it is often 512bytes

Numeric user and group ids and user and group names can both be used with the
-user and -group conditions

find : Examples

find : Examples
$find / inum 767
Find all files who share the same i-node
i.e hard links
$ find / name core.0 exec rm {}\;
Find all core files and remove them
i.e. remove all crash dumps
$ find / size +400
Find all files larger than 400 blocks
We have previously seen that we can find all files whose listing contains the string
morpheus with by using grep on the output of ls -l
Similar, but more refined, searches can be performed using find.
e.g.
To find all files in the current working directory owned by morpheus use:
find . -user morpheus
To find all files in the current working directory whose group ownership is
morpheus
find . -group morpheus
To find all files called morpheus
find . -name morpheus

Regular Expressions

Regular Expressions
Regular Expressions - regexp - are a powerful
method for finding things
A Regular Expression is pattern of characters used to
match against the same characters in a search.
They usually include meta-characters, which
represent things other than themselves, to refine the
search.

Some of the most powerful UNIX utilities, such

as grep, sed and awk, use regular expressions

The command evaluates text against the pattern to determine if the text and the
pattern match
If they match, the expression is true

Regular Expressions : Meta-characters

Regular Expressions : Metacharacters

Character patterns
.
any character, except newline
[ ]
a range of characters
[^ ]

a range of characters to exclude

Modifying meta-characters
*
0 or more of the preceding pattern
+

1 or more of the preceding pattern

exactly one of the preceding pattern

escapes following character

Positional meta-characters
^

anchors match to start of line

anchors match to end of line

Regular Expressions are built up from text and meta-characters.

Common meta-characters and their meanings are listed in the slide.
Not all meta-characters can be used with all commands that use regular expressions.
The backslash - \ - character works in two ways.
If a character has a special meaning, then preceding it with a backslash will cause it
to be matched as the literal character.
i.e. to match a literal dot, the Regular Expression would include \.
The backslash can also be used to give a special meaning to an otherwise normal
character.

Regular Expressions: Character Classes

Regular Expressions: Character

Classes
Class
[:almun:]
[:alpha:]
[:blank:]
[:cntrl:]
[:digit:]
[:graph:]
[:lower:]
[:print:]
[:punct:]
[:space:]
[:upper:]
[:xdigit:]

Matching Characters
Printable characters (including w hitespace)
Alphabetic characters
Space and tab characters
Control characters
Numeric characters
Printable and visible (non-space) characters
Lowercase characters
Printable characters (includes w hitespace)
Punctuation characters
Whitespace characters
Uppercase characters
Hexedecimal digits

These character classes are defined by the POSIX standard and may or may not be
available in commands which implement Regular Expressions.

Regular Expressions : Two warnings

Regular Expressions : Two warnings

Regular Expressions use similar meta-characters
to the shell
However, not all have the same meaning in both
cases.

The shell will expand bare meta-characters it

recognises before passing them to the
command
To be sure your Regular Expression is interpreted in
the way you expect, it is best to quote it
- .*

- .*

will be expanded by the shell to a list of all

hidden files in the current working directory
is a Regular Expression meaning 0 or more
of any character except newline

Regular Expressions : Examples

Regular Expressions : Examples

\$

Finds the pattern of a dollar sign preceded by at least a

space
.\$

Finds the pattern of a dollar sign preceded by at least one

character
^The

Finds the pattern of a line which starts with the word

The
End$

Finds the pattern of a line which ends with the word End

Translate [tr]

Translate [tr]
tr [set1] [set2]
Translates the characters in set1 to set2
- Each input character found in the range specified by
set1 is replaced by the character in the same relative
position in the range specified by set2.
trinity$ cat dante | tr [a-z] [A-Z]
HALF WAY ALONG THE ROAD WE HAVE TO GO,
I FOUND MYSELF OBSCURED IN A GREAT FOREST,
BEWILDERED, AND I KNEW I HAD LOST THE WAY.
IT IS HARD TO SAY JUST WHAT THE FOREST WAS LIKE,
HOW WILD AND ROUGH IT WAS, HOW OVERPOWERING;
EVEN TO REMEMBER IT MAKES ME AFRAID.
...

The tr command takes two ranges and replaces every character in the input that
matches the characters in the first range, by every character in the same relative
position in the second range.
The POSIX character classes mentioned previously, are particularly useful with the
tr command.
For example, the translation shown in the slide, could be rewritten as follows
trinity$ cat dante | tr [:lower:] [:upper:]

Finding things [grep]

Finding things [grep]

$ grep [-i] [-v] [-c] pattern filename
Global Regular Expression Patter n matcher
finds pattern in filename
[-i]

ignore case

[-v]

match ever ything but patter n

[-c]

counts matches
trinity$ grep root /etc/passwd
root:x:0:0:root:/root:/bin/sh

Finds the patter n root in /etc/passwd

trinity$ grep "^May 13" /var/adm/messages | grep -c warning
3

Counts how many warning messages were logged on May 13

The grep command is particularly useful for finding simple strings in files or
output.

Stream Editor [sed]

Stream Editor [sed]

$ sed [option] file
Editor for files
[option] take a look at the man page
Common usage is to substitute a pattern for another pattern
sed -e s/pattern1/pattern2/flags file

trinity$ cat infile

Truman was a Democrat President of the United States of America.
trinity$ sed -e 's/Truman/Clinton/g' infile
Clinton was a Democrat President of the United States of America.

Substitutes all occurrences of Truman with Clinton in infile

and send the changes to stdout

sed : Substitution Examples

sed : Substitution Examples

sed works on a file one line at a time
By default, only the first match on a line is
substituted
s/yes/no/

substitutes the first occurrance of yes on a line with

the string no

To match all occurrences of a pattern, use the

g flag
s/yes/no/g

substitutes all occurrances of yes

Another useful flag that can be used with substitutions is:
n
a number (between 1 and 512) indicating that the replacement
should occur for only the nth
occurrence of the pattern

sed : Substitution Examples

Leaning Toothpick Syndrome
s/\/usr\/bin\/ksh/\/usr\/bin\/sh/
often occurs when searching for unix paths
can be avoided by using an alternative pattern delimiter, for
example :
s:/usr/bin/ksh:/usr/bin/sh:
if the delimiter still occurs in the pattern it must be escaped

Delimiters occur three times in each substitution

statement
Once to start the expression, once between the two patterns
and at the end to close the expression

The name Leaning Toothpick Syndrome (LTS) was coined by Larry Wall, the
creator of the perl programming language, which makes heavy use of Regular
Expressions in its syntax.

sed : Substitution Examples

Deleting characters
s/[a-z]//g
s/[^a-zA-Z0-9\-]//g
finds all occurrences of the ranges and subsitutes them with
nothing, i.e. deletes them
[a-z]
- any lower case alphabetic character
[^a-zA-Z0-9\-]
- any character that isnt an upper or lower case alphabetic,
numeric or hyphen character
- removes whitespace
- hyphens have a special meaning within ranges, so need to
escape the hyphen

sed : Substitution Examples

Substrings

\( \) used to create a substring pattern

\n
references the nth matched substring
where n is a single digit
s/^\(DAEMON_OPTIONS.*\)/dnl \1/
s/dnl \(define(\`SMART_HOST\)/\1/

Matches a line and adds dnl to the front of the

line
Matches a line and removes dnl from the front of
the line

sed

sed
sed is an extremely powerful tool for noninteractive editing
particularly useful in scripting

Common usage is to substitute strings in text

files
By default, sed outputs to stdout
can test substitutions without changing file on disk

Handy one liners for sed can be found here:

http://sed.sourceforge.net/sed1line.txt

Who is using the system ? [who]

Who is using the system ? [who]

$ who [-H] [-q] [ am i] [-r]

Display who is currently logged into the system

[-H]
put a header line on the display
[-q]
just list the users
[ am i]
tell me who I am [Real User-Id]
[-r]
shows the current run level
trinity$ who -H
NAME
LINE
morpheus console

TIME
May 24 10:17 trinity
pts/4
May 24 17:36 (trinity.matrix.com)

trinity$ who am i
morpheus
trinity$ who -r
.
run-level 3

May 15 10:28

The who command is useful for the administrator to find out which users are logged
into a system and from where.
The who am i version of the command is useful for finding the real UID of the
current user

Quicker who [w]

Quicker who [w]

$ w [-h] [username]
Display who is currently logged in
[-h]
suppress the header line
[ username ]
just show username

trinity$
12:52pm
User
root

w
up 59 day(s), 20:57,
tty
login@
pts/0
12:52pm

w is similar to the who command

1 user, load average: 0.00, 0.00, 0.01

idle
JCPU
PCPU what
w

Find a spelling [look]

Find a spelling [look]

$ look word
Checks the system dictionary for a word
- /usr/share/lib/dict/words
trinity$ look fred
Fred
Freddie
Freddy
Frederic
Frederica
Frederick
frederik
fredricite

is a dedicated command for searching the system dictionary which is usually

located in /usr/share/lib/dict/words
The system dictionary is a plain text file, so is also searchable with grep
look

Processes and Control

Processes and Control

Careful Planning is the key

to safe and swift travel

Ulysses

Process Overview

Process Overview
All tasks within Unix begin as processes
Processes are running files
Process Identification Numbers (PIDs) are used
to keep track of separate processes
All processes have a UID and a GID associated
with them
One process may have many Light Weight
Processes, or threads, associated with it

Generating Processes

Generating Processes
Parent process runs fork() to generate an identical copy
of itself except it is given a new process ID
If fork() succeeds
child process inherits parents envir onment space
STDIN, STDOUT and STDERR redirected to parent
fork() returns value of child PID to parent
uses exec() to r un command
when command finishes, returns ONLY the exit status of
command ($?) to parent

If fork() fails
returns exit status of -1 to parent

Generating Processes
STDERR

PARENT
STDIN

STDOUT

ENV

failure
success
(i.e. -1)
Child PID
fork()
fork()
STDERR
Child
NOT created

Incorrect
ENV
CHILD permissions
status
STDIN
Out
ofcode
memory
STDOUT
exec()
($?)

STDOUT
STDIN
Out of process
slots

Understanding Processes

Understanding Processes
The cd command must be built into the shell
Why?
cd changes directory
- i.e. modifies the $PWD environment variable

If cd were an external command, it would follow the

fork()/exec() model
- child processes cannot change the environment space
of the parent
- child processes can only return numeric exit status of
command exec()-ed

The current working directory is defined by the PWD environment variable.

To change directory, this value held by this variable must be changed.
If the cd command were implemented as a command external to the shell, what
would happen?
When the command was invoked, a new process would be created by fork() with a
copy of the parents (i.e. the shells) environment, which would include the PWD
variable.
The code to change the PWD variable would then be exec()-ed in the child process,
and then the command would end.
When the child process ends, it returns only the exit status code of the process that
has been run. So, if the variable had been changed successfully, the parent shell
would receive 0.
Has this affected the PWD variable in the parent process? No. Child processes
cannot affect their parents environment.
So, have we changed directory? No.
Thus the cd command must be an internal process to the shell.

Displaying the Processes [ps]

Displaying the Processes [ps]

$ ps[-e] [-f] [-u username]

Displays the processes running on a host

[-e]
every process
[-f]
full listing
[-u username]
list the processes with UID of
username
$ ps ef | more
UID
PID PPID
root
0
0
root
1
0
root
2
0
root
3
0
morph 126
1
---More---

C
80
80
27
80
80

STIME
16:46:40
16:46:40
16:46:40
16:46:40
18:13:12

TTY
?
?
?
?
console

TIME
00:01
00:40
00:00
04:33
00:01

CMD
sched
/etc/init pageout
fsflush
ksh

The ps command prints information about active processes.

Without options, ps prints information about processes that have the same effective
user ID and the same controlling terminal as the invoker. The output contains only
the process ID, terminal identifier, cumulative execution time, and the command
name.
Otherwise, the information that is displayed is controlled by the options.

Displaying the Processes [prstat]

Displaying the Processes [prstat]

$ prstat [OPTIONS]
By default, shows an iterating list of processes sorted by CPU
usage
[OPTIONS] can be found in the man page and define
alternative methods of displaying and sor ting processes
trinity$ prstat
PID USERNAME
509 root
504 root
508 root
502 root
62 root
26 root
NPROC USERNAME
40 root
1 smmsp

-a
SIZE
4568K
328K
2576K
1872K
2784K
1120K
SIZE
87M
4312K

RSS
4344K
256K
1888K
1288K
1952K
376K
RSS
55M
1416K

STATE PRI NICE

TIME
cpu7
59
0
0:00:00
sleep
59
0
0:00:00
sleep
59
0
0:00:00
sleep
59
0
0:00:00
sleep
59
0
0:00:00
sleep
59
0
0:00:00
MEMORY
TIME CPU
7.6%
0:00:04 0.1%
0.2%
0:00:00 0.0%

CPU
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%

PROCESS/NLWP
prstat/1
sh/1
bash/1
in.telnetd/1
picld/4
dhcpagent/1

Total: 42 processes, 104 lwps, load averages: 0.00, 0.00, 0.01

prstat is a useful way to see which commands are using the most CPU. This can be
particularly useful if a machine appears loaded unexpectedly.
It is also useful to see which users are using a machine most intensively.
prstat has many options which are listed in the man page
The -a option shown in the slide, shows user information at the same time as
process information

prstat : column headings

prstat : column headings

Column headings read as follows

process ID
username of process ow ner
total virtual memory size of process in kilobytes
(K), megabytes (M) or gigabytes (G)
Includes the heap and stack assigned to the
process
RSS
resident set size, as above without the heap and
stack
state
current state of the process
- cpun
- running on cpun
- sleep - process is sleeping
- run
- process is runnable
- zombie - process is terminated but parent has not waited
for return value
- stop
- process is stopped
PID
username
size

A zombie process is a process that has completed execution but the exit status of
which has not been read by the parent process yet.
A process becomes a zombie process when it issues the exit subroutine and the
following circumstances occur: Its parent process is not running a wait subroutine
and has not notified the operating system that it does not intend to wait for its
children to finish.
The it is said that process has died, but has not yet been reaped.

prstat : column headings

pri

process priority

- most user processes run with priorities between 0 and

59
- OS processes run with priorities between 60 and 99
- real time processes run with priorities between 100
and 129 (these are extremely rare)

nice
time
cpu
process
NLWP

NPROC

a modification value for process priority

time running on CPU
percentage of CPU used
name of process
number of Light Weight Processes
(threads) associated with process
number of processes associated with a
user

Nice values range from 0 to 39 and are used to bias the priority of a process. A
higher number results in a lower priority.
There are commands to change nice values, however, users can only affect the nice
value of processes which they own. They also cannot start processes with nice
values less than 20, nor can they lower the nice values of their processes after
they've raised them.

Managing Processes

Managing Processes
Commands executing take over the current
display until they complete
This is called foreground processing

In general, we want the ability to have many

commands running at once without needing a
separate console for each one
Command executed in the background leave the
current display free to run others jobs

Processes exist in one of three states: background, foreground, or stopped.

By default, all processes run in the foreground. However, it is possible to start
processes in the background or change processes from foreground to background
processing.

Background a command [ & ]

Background a command [ & ]

$ command [options] [parameters] &
Execute the command with its options and parameters in the
background
trinity$ find . name *.gif &
[1] 3342 &
trinity$

The command will be placed in background for execution, and a

new command may be issued immediately on retur n of the
system prompt whether or not the first command has completed
3342 is the process-id of the job.
When it is finished, [DONE] 3342 will be displayed

A command run with a trailing ampersand (&) will be executed in the background.
It will be assigned a job number, separate from its process ID and both the job
number and process ID will be printed to the display.
When the command has finished running a done message and the process ID are
printed to the display on which it was started.
If the controlling terminal is killed, all running jobs in that terminal are also killed.
The terminal is the parent process for the jobs. When a parent process is killed, all
its children are also killed.

nohup

nohup
$ nohup command [&]
When a command is invoked by the nohup command
it will ignore hup signals
In practice, this means that the command is not
killed if the parent terminal dies
nohup can also be applied to processes after they
have been started using the -p PID option
trinity $ nohup find / -name bin
Sending output to nohup.out

nohup is a simple shell command that tells the program it is told to run not to exit
when the controlling terminal is killed.
It is particularly useful for leaving long running processes in the background
without having to keep a shell open - i.e. keeping them running in case you need to,
or accidentally, log out (or if the windows computer on which you've logged in
freezes and needs to be restarted.)
Example:
nohup long_running_command &

The '&' tells the shell to not only nohup the command, but to also run it in the
background, letting you continue using the terminal as usual.
By default, nohup sends all output (both stdout and stderr) to a file called nohup.out
in the current working directory. However, if this directory is not writable, the file
will be created in the users home directory. This file can be very useful for
examining if there are problems with the job.

Listing the jobs [jobs]

Listing the jobs [jobs]

$ jobs[-l]
Displays the processes which were placed in the
background from this shell
[ -l ] display the process ID and not just the
job-id

trinity$ jobs
[1] + Running find . name *.gif &

will only show the jobs started in the terminal in which it is run. This can be a
problem if you have started jobs in multiple terminals or have used nohup and
closed the controlling terminal.
jobs

Foreground a command [fg]

Foreground a command [fg]

$ fg %job-id
Moves the process running with job-id from the
background to the foreground

trinity$ jobs
[1] + Running find . name *.gif &
trinity$ fg %1

Processes running in the background may also be stopped with stop %job-id
Stopped jobs may be restarted with bg %job-id

Killing processes [kill]

Killing processes [kill]

$ kill [-signal] [%job-id | PID]
Ends a processes specified by either job-id or PID
[-signal]
how badly you want it to end
[see man s 3HEA D signal ]

$ jobs
[1] + Running find . name *.gif &
$ kill %1
[1] + Terminated find . name *.gif

Ends a processes giving it it chance to close files

$ kill 9 %1
[1] + Terminated find . name *.gif

Kill a process with extreme prejudice. Just end.

The default signal sent by kill is 15, SIGTERM.

Useful signals include:
Name Value
DefaultEvent
SIGHUP
1
Exit
Hangup - causes some programs to
re-read their configuration files
SIGKILL
9
Exit
Killed
SIGTERM
15
Exit
Terminated - exits gracefully
One sometimes sees programs fail with the following signals:
SIGSEGV
11
Core
Segmentation Fault - program dies
and produced a core dump
SIGPIPE
13
Exit
Broken Pipe - a command in a
pipeline failed, breaking the pipeline

Job Management Commands

Job Management Commands

Command

Function

jobs

Display w hich jobs are current running

fg %n

Foreground a job

bg %n

Background a job

kill %n

Abort a job

Ctrl-C

Abort the current job

Ctrl-Z

Suspend the current job, ready to be

sent into the background

Job management commands only show or operate on jobs started in the terminal in
which they are run.
It is not possible to background a process in one terminal and then foreground it in
another.

/proc

/proc
Special filesystem
Contains information about all running processes
memory segments (as)
current w orking director y (cwd)
light weight processes (lwp)

ps and prstat are ways of displaying some of the

contents of the /proc filesystem
Most files within /proc are plain text and can be read
by the user, however, there is generally no explanation
of the format

/proc is a file system that provides access to the state of each process and lightweight process (thread) in the system. The name of each entry in the /proc directory
is a number corresponding to a process-ID. These entries are themselves
subdirectories.
Access to process state is provided by additional files contained within each
subdirectory.
The owner of each /proc file and subdirectory is determined by the user-ID of the
process.

vi: Visual Editor

vi: Visual Editor

Tell you what. Let me

sweeten the deal for you
a bit

Beelzebub

The Visual Editor

The Visual Editor

A screen oriented text editor with many
enhancements to greatly speed up the entry of
program source files
Well adapted to and created for the C language
At best, very powerful but very cryptic in use of
commands
At worst, an abomination from the sons of dogs
Often, the only editor available on a system

Modal Editor

Modal Editor
Two fundamental modes of operation
Insert/Replace Mode

- for entering new text or overtyping text

Command Mode

- for all other functions

deleting text, moving cursor, saving work, copying
text, etc.

Editing a file

Editing a file
To edit a file using the vi editor, just type
$ vi filename

To enter the vi editor without specifying the file

name in advance, type:
$ vi

Leaving vi

Leaving vi
To exit, you must be in Command mode
To verify this mode, press <ESC> until you hear
a beep

:q!
:w
:wq
ZZ

exit immediately and discard all changes

save file only, remain in vi with current file
write the file and exit back to the shell
write the file, and exit back to the shell

Entering Text

Entering Text
To enter insert mode from Command Mode,
press
i insert text before the cursor
a append text; insert after the cursor
o insert a new line after cursor line and remain in
insert mode
O insert a new line before the cursor line

To exit back to command mode, press the

<ESC> key

Insert / Replace mode

Insert / Replace mode

As you type, follow each line by pressing the
<Return> key
The <backspace> key, denoted by the
following:
moves cursor back and erases the previously typed
characters, while remaining in insert mode

Cursor keys may not do what you expect while

in this mode

Command Mode

Command Mode
To verify you are in Command mode, press
<ESC> until you hear a beep
Two forms of commands
direct commands

- executed immediately

ex commands
- are always preceded by a colon :
- followed by the command

Cursor Modes

Cursor Modes
The cursor keys may move one character in the
direction of the arrow
More certainly, you should use the following
keys:

j
k
h
l
space

move cursor down one line

move the cursor up one line
move the cursor to the left on character
move the cursor to the right one character
move the cursor to the right one character

Moving around in vi

Moving around in vi
To move further, you may use the following
keys:

Ctrl-f
Ctrl-b
G
3G
1G
0
$

move forward one screen

move backward one screen
move to end of the file
move to line 3 of the file
move to line 1 of the file
move to the beginning of the line
move to the end of the line

Deleting Text

Deleting Text
Removing text by the cursor:

dd
5dd
d$
dw
x
5x

deletes the cursor line

deletes the current line and 4 more
deletes from the cursor to the end of the line
deletes from cursor to next white space
deletes the character under the cursor
deletes the character at the cursor and 4
more
undoes the last change, and only the last.

Searching in vi

Searching in vi
Text can be searched in the following ways:

advance the cursor from present

position to the next occurrence of
pattern
?pattern
advance cursor from present position
to previous occurrence of pattern
n
repeat the previous search in same
direction as original search
N
repeat the previous search but in the
opposite direction to original search
%
Find the matching delimiter - with
cursor on one of the characters {, [, (
find the matching },], ) or vice-versa

/pattern

Replacing Text

Replacing Text
To replace text from Command Mode, enter one
of the following:
replace the character under the cursor with
the letter d. No <ESC> necessary
R
global replace mode. Text under cursor will
be replaced with typed text until <ESC> is
pressed
cw
Change the word from current cursor until
and not including the next white space with
the typed text until <ESC> is pressed.
A $ symbol marks the end of the text to be
replaced
rd

More text replacement

More text replacement

Modifying text continued:
3cw
C$ or C
~

change the next three words

change the text from the cursor until end
of line
map uppercase to lower or lower case to
upper for the character under the cursor

Substituting text

Substituting text
:s/first_exp/replacement_exp/[g]
Substitute the first_exp with the replacement_exp
The [g] at the end means global to the line.
:1,$s/first/second/ or :%s/first/second/
For the entire file, replace the first occurrence in a line of the
string first with the string second
:.,$s/meat/bones/g
From the present cursor line to the end of the file, substitute all
occurrences of the string meat with the string bones

Coping text

Coping text
Text can be copied with the following
commands
yy yank (copy) the current line to the temporary
buffer
3yy yank the current line plus two to the
temporary buffer
xp delete the character under the cursor and
retrieve the deleted character after the cursor.
(Interchanges two characters)
J join the current line with the next line

Retrieving text

Retrieving text
Yanked and deleted text can be inserted using:
pull (insert) the contents of the temporary buffer
following the cursor
P pull the contents of the temporary buffer before
the cursor
p

Miscellaneous tricks

Miscellaneous tricks
A few extras:
Ctrl-l
Ctrl-r
Ctrl-G

Redraw the screen contents

(lowercase L)
Redraw the screen contents
File status

ex commands

ex commands
ex commands in fact invoke the ex line editor
on the file
for more information, see man ex

All ex commands
are executed from Command Mode
are prefaced by :
are terminated by a carriage return

Range of command may be restricted through

line numbers and patterns

! (bang)

! (bang)
:q!
In this context the command is to be executed
without error reporting

:!ls
In this context, it means to execute and display the
results of the shell command ls

:r !ls
Perform the shell command ls and insert the results
into the present file following the cursor

set commands

set commands
The set command allows modification of the
environment
The environment variables which can be
accessed through the set command can be seen
through the :set all command

Typical Set Commands

Typical Set Commands

:set ai
Auto indent; causes the cursor to automatically indent
to match the previous line in Insert Mode
:set noai
no-auto-indent; turns ai mode off
:set nu
Number; causes line numbers to be displayed
:set nonu
no-number; causes line numbers to be turned off
:set sm
showmatch: causes a momentary display of the
matching delimiter whenever a delimiter [], {}, ()
is typed. Very handy in C

.exrc : making set commands stick

.exrc : making set commands stick

vi customisation start-up file checked for in the
current working directory, then $HOME directory
trinity$ cat ~/.exrc
set ai sm tabstop=4

enable auto-indent mode

enable showmatch mode
set tabstop to 4 spaces

Final Thoughts

Final Thoughts
All work is done on a temporary file
No changes to the original file occur until a
command which causes the write has been
executed
User must have write permission to modify the
file
User needs only read permission to view the file
All files may be edited, but binary files are
unintelligible

Shells and Variables

Shells and Variables

One cannot collect all the

beautiful shells on the beach.
One can collect only a few, and
they are more beautiful if they
are few.
Anne Morrow Lindbergh

What is an Environmental Variable ?

What is an Environmental Variable ?

A variable is a placeholder for information to be
used by a process
There are two categories of variables
Local (shell)
Global (environmental)

Environmental variables include

PATH controls where to look for commands to
execute
PWD absolute path of current working directory
i.e. defines where you are currently in the file
hierarchy
TERM describes the type of terminal in use
Other environmental variable hold information about the current user, the name of
the machine, the local time zone, the machine architecture, process ids and the exit
status of the last command executed.

Where do they come from ?

Where do they come from ?

From the shell
Initially, when a variable is created, it is local
and so only available to the shell of its origin
To make it global, it must be exported
Can be customized or pre-defined by the user
By convention, Bourne and Korn shells use
capital letters for pre-defined, built-in variables.

Example variables : Which shell?

Example variables : Which shell?

Look at the prompt
$
%
#

Standard Bourne Shell (sh), Korn Shell (ksh)

Standard C Shell (csh)
This prompt is usually reserved for the root
(Super-User)

Examine the shell environmental variable

$SHELL
$shell

sh, ksh, bash

csh

sh is the standard command interpreter for Unix systems.

ksh, the Korn shell, csh, the California shell and bash, the Bourne again shell are
alternatives that extend the functionality of the shell environment.
ksh is the default user shell in Solaris.

Example variables : Process ID [$$]

Example variables : Process ID [$$]

When scripting, it is often useful to know the Process ID
of the script itself
Useful for naming temporary files

$$

variable contain the PID of the script being run

When invoked in a shell, it gives the PID of the shell

trinity$ cat testscript.sh
#!/bin/sh
echo the PID of this script is $$
trinity$ sh testscript.sh
the PID of this script is 3829
trinity$ echo $$
417
trinity$ ps ef | grep 417
root
417
413 0 13:02:51 pts/3

0:00 ksh

Example Variables : Exit Status [$?]

Example Variables : Exit Status [$?]

Numerical value that indicates whether an
executed command was successful
Zero Value
= Successful
Non-Zero Value = Not Successful

$?

Exit status of last executed command

trinity$ touch newfile
trinity$ echo $?
0
trinity$ mkdir newfile
mkdir: Failed to make directory newfile"; File exists
trinity$ echo $?
2

Creating local variables

Creating local variables

$ variable_name=value
Creates a local variable variable_name and assigns
to it the value value
NOTE: there are no spaces around the =
NOTE: this only affects the current shell
trinity$ ZERO=1
trinity$ echo $ZERO
1

Variables can be viewed with the echo command and the name of the variable,
prefaced by a $ symbol to denote to the shell that it should treat the name as a
variable name and expand it before displaying it.

Removing environmental variables

Removing environmental variables

$ unset variable_name
removes variable from the environment

trinity$ unset ZERO

trinity$ echo $ZERO
trinity$

It can be problematic to remove some built-in variables. For example, without a

PATH variable you will have to type the full path to every command you want to
run.

Making variables global [export]

Making variables global [export]

$ variable=value; export variable
Creates a variable variable and assigns to it the
value value
Then, places it within the global environment
available to this shell and any child processes of this
shell
Note: there are no spaces around the =
trinity$ ZERO=1; export ZERO

Export should be used with the bare variable name. A common mistake is to use
export $variable which the shell will expand to the contents of the variable
before attempting the export.

Displaying the environment [set] [env]

Displaying the environment [set]

[env]
$ set
Displays the current shell variable space (local)

$ env
Displays the current environment space (global)
trinity$ set
ERRNO=1
FCEDIT=/bin/ed
HOME=/
IFS='
'
LINENO=1
LOGNAME=root
MAILCHECK=600
OPTIND=1
PATH=/usr/bin:/bin:/usr/sbin:/sbi
n

Some variables appear in both listings.

trinity$ env
_=/usr/bin/env
SSH_TTY=/dev/pts/1
PATH=/usr/bin:/bin:/usr/sbin:/sbi
n
LOGNAME=root
USER=root
SHELL=/sbin/sh
HOME=/
SSH_CLIENT=172.16.0.100 34415 22
TERM=screen
PWD=/

Search paths [PATH]

Search paths [PATH]

$ PATH=directory [ :directory... ]
$ export PATH;
The shell uses the PATH variable to locate commands
in directories
Directories are searched in the same order as they
are specified in the PATH statement
Each directory is searched for the existence of the
command which is to be executed
The first command found is executed

Search paths may contain an entry for dot, i.e. the current working directory. This
is considered unsafe for users and dangerous for the super-user as it can allow the
running of arbitrary code by accident.

Searching the PATH [which]

Searching the PATH [which]

$ which command
The which command displays the pathname leading
to an accessible command based on your search path
If a particular command isnt found, you may need to
modify the PATH variable in order to add the
directory in which it is located
trinity$ which passwd
/usr/bin/passwd

which takes a list of names and looks for the files which would be executed had
these names been given as commands.
Each argument is expanded if it is aliased, and searched for along the user's PATH.
Both aliases and PATH are taken from the user's shell.

Making the environment stick

Making the environment stick

Initialisation files
System-Wide
/etc/profile
/etc/.login
User specific, in $HOME
~/.profile
~/.kshrc
~/.login
~/.cshrc

It is sometimes useful to create shell variables which are available every time a new
session is started without needing to be hand set.
e.g. in an Oracle environment, the ORAHOME variable is commonly set to last
between sessions.
In general, it is good practice to leave global shell configuration files as set up by
the operating system vendor and make changes only in the user-specific files.

Initialisation file sequences

Initialisation file sequences

shell

on login

on new shell

sh

/etc/profile
$HOME/.profile

csh

/etc/.login
~/.cshrc
~/.login

~/.cshrc

ksh

/etc/profile
~/.profile
~/.kshrc

~/.kshrc

bash

/etc/profile
~/.bash_profile
or ~/.bash_login
or ~/.profile

~/.bashrc

Typically the ~/.bash_profile contains

if [ -f ~/.bashrc ];
then source ~/.bashrc;
fi
as the last entry to get ksh like behaviour

More about Shells

More about Shells

Try to be like the turtle at ease in your own shell.

Bill Copeland

Quoting in Shells [ ]

Quoting in Shells [ ]
Use single quotation marks to identify text literally
trinity$ echo the path is $PATH
the path is $PATH

Use double quotation marks to identify text literally,

while enabling variable and command expansion
trinity$ echo the path is $PATH
the path is /bin:/sbin:/usr/ucbbin

Backticks, or graves, ` `, force the command enclosed

in them to be executed and its output substituted into
command before the remainder of the line is evaluated
trinity$ echo the hostname is uname -n
the host name is trinity

Why quote?
We have seen that the shell has meta-characters which have special meanings. We
have also seen that some of these characters have meanings to commands. We can
use quoting to force meta-characters to be dealt with in the way we expect, and by
the command we expect.

Escaping in shells [\]

Escaping in shells [\]

trintiy$ echo \the path is \$PATH\
the path is $PATH

The back-slash \ escapes the the meaning of the

following character so that it the shell does not
interpret it, if it normally would

The backslash - \ - allows single characters to be escaped so that they are not
interpreted by the shell. This can be particularly useful for escaping quotes or
single characters that would still be interpreted inside quotes.

Custom Prompts

Custom Prompts
$ PS1=value
PS1 is a predefined prompt variable that
users can change
NB there are no spaces around the =
$ PS1=uname -n:\$PWD $
trinity:/home/morpheus $

It can be useful to set the prompt in your shell to display certain information.
For example, if you are connected remotely to many machines, having the name of
the machine in the prompt can help identify which session is which, and reduce the
possibility of running a command on the wrong machine.
It may also be useful to display the current working directory in the prompt, so that
it is clear where one is in the file hierarchy at any time.
The prompt variable is a commonly set in your personal shell initialisation file.

Aliases [alias]

Aliases [alias]
$ alias [name=value]
with no options it lists which commands have aliases
substitutes the name for value when executing
commands so that the command set can be richer
NB there are no spaces around the =
trinity$ alias
trinity$ dir
-rw-r--r-1
drwxr-xr-x
2
-rw-r--r-1

dir=ls l
morpheus
morpheus
morpheus

users
users
users

257 Apr 18 10:51 dante

117 Apr 18 10:43 newdir
0 Apr 18 10:43 newfile

A popular use of aliases is to provide a way of using familiar commands on new

systems, however, these need to be recreated on each new machine that is used and
can cause problems if the vendor changes the way the command works between
versions.
Aliases can also be dangerous, if they change the way a command works or which
command is run without showing this to the user.
In general, aliases should be avoided.

Removing Aliases [unalias]

Removing Aliases [unalias]

$ unalias name
removes name from the alias tables
trinity$ unalias dir
trinity$ dir
ksh: dir: not found

Command recall [history]

Command recall [history]

$ history
Recalls the last commands
$HISTSIZE
limits the size of the history
$HISTFILE
determines where the history is stored

trinity$ history
28
man set
29
man which
30
man -k which
31
dir
32
history
trinity$

When typing long commands, or pipelines, it can be useful to be able to recall them
to run again.
The history command lists the most recently run commands.

Command line editing: the vi way

Command line editing: the vi way

$ set o vi
Allows you to recall command, re-execute
them and edit them using vi commands
-

j
k
h
l
space

move
move
move
move
move

cursor down one line

the cursor up one line
the cursor to the left on character
the cursor to the right one character
the cursor to the right one character

trinity$ set o vi
press <esc> key and then k until the desired command
is displayed
press <CR> to execute that command

ksh offers two methods of command line editing, using keys borrowed from the two
most popular Unix editors.

Command line editing: the emacs way

Command line editing: the emacs way

$set o emacs
Allows you to recall commands, re-execute them and
edit them using emacs commands
-

Ctrl-n
Ctrl-p
Ctrl-f
Ctrl-b
Ctrl-d
Ctrl-a
Ctrl-e

move down one line

move up one line
move right one character
move left one character
delete a character
move to start of line
move to end of line

trinity$ set o emacs

press <ctrl-p> until command is displayed
press <CR> to execute command

Emacs command editing mode may be preferred by people who are more used to
the emacs text editor.

Shell Scripting

Shell Scripting
I write scripts to serve as
skeletons awaiting the flesh
and sinew of images.

Ingmar Bergman

What is a command ?

What is a command ?
A command is a file which is marked executable
May be a pre-compiled binary or a script

If you have execute permissions on the file,

then you may run it
Remember the fork()/exec() model

Commands may also be a functions built into

the shell
e.g. cd. alias, history

Shell Scripts

Shell Scripts
What is a script ?

A file which is mar ked executable whose first line is of the form

#!/path/to/command
The #! header tells the operating s ystem w hat s ort of
commands are in the script

Why script?
A means of automating tasks

Shell scripts can be

Simple
Complex

#! - can be pronounced as hash-bang, sh-bang, or hash-shriek

Flow control

Flow control
Commands within a script are usually executed
one after the other in the order they appear in
the file
Flow control commands can use input data to
decide which command to execute next
test condition or [ condition ]

Examples of flow control commands include

if / else / elif
until / while
for

Flow control allows scripts to be more than just a list of commands, by allowing
branches in logic based on conditions and data.

Testing conditions [test]

Testing conditions [test]

The test command is used to test conditions
It can be used in two ways
test [condition]

[ [condition] ]

Commonly used conditions include

file tes ts
- -d path
tests if the path supplied is a director y
- -x path
tests if path is an executable file
integer comparisons
- -eq and -ne equal to and not equal to
string comparisons
- = and !=
is the same as and is not the same
as

The test utility evaluates the condition and indicates the result of the evaluation by
its exit status.
An exit status of zero indicates that the condition evaluated as true and an exit
status of 1 indicates that the condition evaluated as false.
When comparing strings and variables in scripts, it is a good idea to place them in
quotes.

Flow control: if

Flow control: if
Used to test the status of a condition and
proceed with an action of the status of the
condition is true
if [ condition ]
then
command
fi
fi marks end of if block.
Square brackets [ ] are used as shorthand for the
test command
- So, must have a space after the [

Flow control: else

Flow control: else

Creates a two-way branch
perform one action if the status of the condition is tr ue
perform another action if the status of the condition is false
trinity$ cat elsetest.sh
#!/usr/bin/bash
if [ $LOGNAME != neo ]
then
echo Go away, $LOGNAME. You are not The One.
else
echo Hurrah. You are $LOGNAME. I have found The One!
fi
trinity$ who am I
morpheus
pts/2
Apr 19 20:20
(trinity.matrix.com)
trinity$ ./elsetest.sh
Go away, morpheus. You are not The One.
trinity$ su neo
Password: ********
trinity$ ./elsetest.sh
Hurrah. You are neo. I have found The One!

Flow control: elif (i.e.else if)

Flow control: elif (i.e.else if)

Offers a second condition if the previous is
false
if [ condition1 ]
then
command1
elif [ condition2 ]
then
command2
else
command3
fi

Flow control: nested if statements

Flow control: nested if statements

if statements can be nested
This allows multiple conditions to be tested
Offers a second test, if the previous one is true
if [ condition1 ]
then
if [ condition2 ]
then
echo Both conditions are true.
fi
fi

Flow control: while / until

Flow control: while / until

while and until statements are useful for
looping until a particular condition occurs.
This method allows flexibility if the number of tasks
repeated (such as a for loop) is uncertain.

The while command tests and executes a

series of commands as long as a condition is
true.
The until command continues the loop until a
true condition exists.

The until construct is not available in csh.

Flow control: while / until examples

Flow control: while / until examples

trinity$ sh while.sh
number is 0
number is 1
number is 2
number is 3
number is 4
number is 5
number is 6
number is 7
trinity$

num=0
while [ $num -lt 7 ]
do
echo number is $num
num=`expr $num + 1`
done

password=today
name=nobody
until [ $name = pass
do
echo Enter Pass
read name
done

trinity$ sh until.sh
Enter Pass fred
Enter Pass bill
Enter Pass password
Enter Pass pass
trinity$

In the first example, the loop continues while the condition is true, i.e. until the
condition is false.
In the second example, the loop continues until the condition is true, i.e. while the
condition is false.
The two contructs can be used interchangeably, depending on which is the easier
form of logic to express.

Flow control: for

Flow control: for

The for statement is used to process a list
operations are repeated for a list of known or fixed
values
loop continues until each value has been processed
arguments are either sourced in script, or stated
directly after for statement.

for arg in [list]

do
command(s)...
done

Use the for command to iterate through a know list of arguments.

Positional Parameters

Positional Parameters
Data can be passed to the script from the
command line
These parameters are referenced with the
following variables

$0
$1 to $9
$#
$*

$*
$@
$@

name of command used to call script

first nine positional parameters
contains number of parameters
all positional parameters as one long
string
put single pair of double quotes
around whole string
same as $*
put double quotes around each
parameter

Data can be passed to scripts from the command line, as with any command.
These parameters are accessed using the positional parameters variable.
$0 is the first item from the command line, i.e. the command used to invoke the
script
$1-$9 are the first 9 parameters passed to the command.
$* is a single string containing all parameters, quoting it, will put quotes around the
whole string.
$@ is a single string containing all parameters, however, quoting it will put quotes
around each parameter in the string
$# contains the total number of paremeters

Positional Parameters: continued

Positional Parameters: continued

Can access beyond $9 by using set braces
e.g. ${10} will access 10th argument

${var} can be used instead of $var

useful if followed by alpha-numerics

$? is the exit status of the last command

0 means successful
Script can be ended with exit n, where n is the
exit status or condition code of the script

Set braces can be used to access beyond the ninth parameter and also to make
variable names clear.

Networking

Networking
The Network is the Computer

S. McNeally

Networking and Distributed Systems

Networking and Distributed Systems

Most computer installations

today are based on networks
allows information and resources
to be shared more easily
applications can be distributed
over multiple systems

Nowadays, typical computer installations consist of multiple systems connected

together using some form of networking technology. It is now possible to construct
highly complex heterogeneous networks, where the computers may be of
different types, running different operating systems and even using different
connection technologies.
For example, it is not uncommon to have networks consisting of a mixture of PCs
running MS-DOS or Windows connected with UNIX based workstation or server
machines and even mainframes. Connection technologies range from simple serial
connections through LAN based methods (Ethernet, Token Ring) to complex
proprietary mechanisms tailored to a particular type of computer.
Networked systems allow individual users on individual systems to share
information more easily - files can be copied between systems rather than
transported on some backup medium. Resources such as printers can also be shared,
allowing them to be used more effectively.
There is also a trend towards building distributed applications, where functionality
is split across multiple systems. For example, a powerful central computer could
store a large database that is accessed from a number of smaller systems that have
less processing power but more sophisticated display and input facilities. This
allows more user-friendly interfaces to be constructed for data input, and for
displaying the results of calculations.

Basic Network Services

Basic Network Services

File Transfer
copying files from one system to another
Remote Ter minal
interactive access to another system
Remote Execution
execution of a program on another system
Remote Printing
sharing printers amongst several systems
Electronic Ma il
sending and receiving messages to/from users on other systems

Basic network facilities:

File Transfer, allowing individual files to be copied from one system to another.
Facilities for this range from very simple mechanisms such as Kermit, through the
UNIX Basic Networking Utilities (uucp) to the sophisticated File Transfer Protocol
(FTP) as provided by TCP/IP
Remote Terminal access, allowing a user at one system to undertake an interactive
session on another. The mechanism is to have the user log in to the remote
system, then the behaviour is exactly as if they had logged in to a local system.
Remote Execution, allowing a user at one system to execute a single program on
another system, without having to actually log in to the remote system.
Remote printing, allowing a user at one system to print a file on a printer that is
attached to another system elsewhere on the network. This usually happens
transparently (without any special actions required on behalf of the user) but the
user may be required to specify the name of the system containing the printer.
Electronic Mail, allowing users on different systems to send messages to each other.
Electronic mail systems are becoming increasingly sophisticated, it is not
uncommon to be able to send voice messages and other multimedia documents in
this way.

Distributed File Systems

Distributed File Systems

Files appear
local here

Systems can
access files on
remote
machines as if
they were local

Files are
actuall y
stored her e

Applications and
users are
unaware of the
location of the
files

The networking facilities described above generally require the user to have some
knowledge of the network, such as which machine they wish to communicate with.
Modern network environments may also provide facilities where the actual
networking takes place at a lower level, and does not require specific knowledge by
the user that the network is being used.
Distributed file systems were the first systems to provide this. In the UNIX
environment, a user has access to files through a single hierarchical file system
which hides the details of which physical (or logical) disk actually contains the files
(unlike DOS, where each logical drive must be explicitly be named and has its own
hierarchy of files and directories). With distributed file systems, this concept is
extended so that the hierarchy on a system may include directories and files which
exist on a different system in the network.
The most common UNIX distributed file system is the Network File System, or
NFS, covered in some detail later in the course. Others are the Andrew File System
(AFS), and the DCE DIstributed File System (DFS).
The concept of transparent access to files on a remote system also forms the basis
of Novells NetWare network operating system.

Distributed Window Systems

Distributed Window Systems

Graphical output can be dis played
on screen
Input taken from keyboard and
mouse

High power system calculations can be done here

No hi-res screens

The main Graphical User Interface systems for UNIX offer more than the ability to
create and manage windows and graphical output on the screen.
The architecture of the X-11 system allows it to operate in a distributed manner in a
networked environment. The application requiring the graphical output, and
keyboard and mouse input need not be running on the same system as the graphics
screen, keyboard and mouse being used. This allows a situation where a computeintensive application can be run on a more powerful machine on the network, which
does not have graphics facilities. The output is transmitted across the network to a
less powerful system which has a graphics display and can display the output.
Applications may be configured so that this division of computation from I/O is
invisible to the user, it will appear as though the application is running on the local,
graphics-based system.

Protocol Layers

Protocol Layers
Application

Application

message

message

s 1 message 1
t
a 2
message 1 2
c
k 3
message 1 2 3

s 1 message 1
t
a 2
message 1 2
c
k 3
message 1 2 3

To manage complexity, divide into layers

Protocols to perform communication between computer systems are generally very

complex. To help manage this complexity, the protocols are divided into a series of
layers, each of which is responsible for handling some aspect of the
communication.
For example, one protocol layer will be responsible for translating the bits of data
from the computers memory into electrical pulses on the connection medium, and
then converting the pulses back to bits at the destination computer. Another layer
could be responsible for ensuring that each block of data is sent to the correct
destination (viewed from a higher level).
The protocols at each layer each communicate conceptually with a peer protocol (ie
a protocol at the same level) on the remote system. In reality, data is passed down
through the layers on the sending system and up through the layers at the receiving
system. At each layer on the sending system, control information is added to the
original message. At the receiving system, the control information is interpreted by
the peer protocol and removed from the message, so that the original message
arrives intact.
A collection of protocols structured in this way is known as a protocol stack.

The OSI 7 Layer Model

The OSI 7 Layer Model

Generic model for describing inter-system
communications
77 Application
Application Provides
Providesapplications
applicationswwith
ithaccess
accessto
tothe
the
OSI
OSIenvironment
environment
66 Presentation
Ensuresconsistent
consistentrepresentation
representationof
ofdata
data
PresentationEnsures
am
among
ongall
allsystems
systems
Management
55 Session
Managementof
ofaadialogue
dialoguesession
sessionbetween
between
Session
two
applications
two applications
44 Transport
Transport Enhances
Enhancesquality
qualityof
ofservice
servicegiven
givenby
by lower
lower
three
threelayers
layers
Ensures
33 Network
Ensuresblocks
blocksof
ofdata
dataarrive
arriveatattheir
their
Network
correct
correctdestinations
destinations
22 Datalink
Blocks
Datalink
Blocksdata
dataand
andensures
ensurescorrect
correct
transm
transmission
issionand
andreception
reception
11 Physical
Defines
Physical
Definesphysical
physicaland
andelectrical
electrical
characteristics
characteristics

The OSI 7-Layer Model

ISO has defined a standard model to describe communications between computer
systems, known as the 7 layer model for Open Systems Interconnection (OSI). This
model is useful as a framework against which to analyse and compare different
protocols. Each of the 7 layers deals with a different aspect of communication
between systems:
1. Physical - describes the physical and electrical standards of the communications
medium, such as voltages, resistances and cabling standards.
2. Data Link - describes how the communications medium is accessed, and how
groups of bits known as frames are sent and received.
3. Network - describes how data is sent from one system to another, including any
routing details.
4. Transport - describes end-to-end communication, independent of any routing.
5. Session - deals with the dialogue between systems.
6. Presentation - ensures that data consistency is maintained between systems,
irrespective of any differences in machine data representation.
7. Application - presents a service to the user.

The TCP/IP Protocol Stack

The TCP/IP Protocol Stack

OSI Layer 5-7

Application
Application

TELNET, FTP

OSI Layer 4

Transport
Transport

OSI Layer 3

Network
Network

IP (ICMP, ARP, RARP)

Data
DataLink
Link

Ethernet, Token Ring

FDDI, serial, etc.

OSI Layer 1-2

TCP, UDP

The TCP/IP Stack

The Internet protocol suite can be described using a stack model similar to, though
simpler than the OSI 7 layer model.
At the heart of TCP/IP is the Internet Protocol IP. This is a connectionless,
unreliable datagram service whose main job is to forward packets from one system
to another. It can be placed at level 3 (Network) of the ISO model.
Above IP come the two transport protocols, whose job is to move information from
one endpoint to another.
TCP, Transmission Control Protocol, is a reliable, connection oriented byte stream
service between two hosts on an internetwork. Considerations of routing, reliability,
sequencing and flow control are dealt with using the facilities of IP and another
Network level protocol known as ICMP (Internet Control Message Protocol).
UDP, User Datagram Protocol, is a datagram based service between two
internetwork hosts. UDP is not reliable.
Higher level network services are known as Applications, and encompass protocols
at layers 5-7 of the ISO model. A number of application services are defined now.
These include file transfer with FTP and TFTP, and remote terminal emulation with
TELNET.
IP is the lowest protocol defined in the Internet suite. No attempt is made to
define data link protocols. This makes the Internet Protocols very flexible. All that
is needed is a connection medium between two systems, and an implementation of
IP that supports that medium. Currently, IP is available over Token Ring, Ethernet,
FDDI, serial links (although these generally have to be high speed to provide
reasonable service) and wide area links such as X.25.

Concepts

Concepts
APP S

APP S
TCP
IP

UDP
ICMP

TCP
IP

UDP
ICMP

This can take 2 forms

Reliable flow
Unreliable flow

The Network Layer handles the transmission of IP

packets across networks
The Transport Layer provides a flow of data between
two hosts for each application
All that the Network layer provides is transmission of IP packets from host to host
across networks. It does not have any knowledge of which applications own which
data, but simply funnels all data for a particular host from one end to another.
The Transport Layer is where we start to break the data down and direct it to and
from applications across networks.

Connectionless protocols

Connectionless protocols
DATA

ER

There is no guarantee of delivery

There is no guarantee on the order in which data
arrives at the other end
Such protocols are unreliable - like IP

Imagine frames of data as postcards. You want to send 10 frames to a friend in

another city. Sending them via an unreliable protocol means that you have no
guarantee that all 10 postcards will arrive at the other end (though, like the postal
service, in general they will), plus you have no guarantee in which order the
postcards will be delivered.
IP works in this way. IP packets can be lost or damaged. There is no mechanism at
the network layer to deal with the notification of losses to the sender, or to request
that the packets are re-sent. That is left up to a higher level protocol.

Connection-oriented protocols

Connection-oriented protocols
DATA

DATA

TWO-WAY COMMUNICATION

No data loss - reliable

Data arrives at the other end in the same order that
it was sent

Consider data exchange as a telephone conversation. First, a connection is establish

- i.e. the number is dialed and connects. Then two-way data exchange takes place.
At all time the data arrives at the other end in the same order that it leaves - your
words dont arrive in a different order, and vice versa. When the transfer
(conversation) is complete, the connection is closed - we hang up.
This is an entirely reliable data transfer, as we cant lose anything that is transferred
from the other end.

Ports + Sockets

Ports + Sockets
APPLICATION PROCESS

SOCKET
INPUT

OUTPUT

PORT

A socket is a pair of data queues for an application

process
A port is the address of a socket

Sockets are usually implemented as an area of memory within the kernel of an

operating system. Incoming data is bound via its port address to a socket, and
thus directly into an application process.
Port addresses are numbers in the range 0-65535

Important Port Numbers

Important Port Numbers

Port
7
13
19
20 & 21
23
25
80
110
119

Service
Echo
Date + Time
Character Generation
FTP
Telnet
Mail (SMTP)
WWW
Mail (POP)
News

Some port numbers have been reserved for

common services across the internet
In general, port numbers below 1024 are reserved for what are called Well Known
Services (WKS). Numbers above 1024 are free for user applications to do with as
they wish.

UDP - Fundamentals

UDP - Fundamentals
User Datagram Protocol (UDP)
Connectionless
Unreliable

Used for
Domain Name Service (DNS)
Routing Information Protocol (RIP)
Real Audio
Network File System (NFS)
Trivial File Transfer Protocol (TFTP)
Network Time Protocol (NTP)
Plus many others
UDP can be used wherever reliability is unimportant and where a stream of data is
not needed - i.e. no frame numbering is required.

UDP - Frame format

UDP - Frame format

2

OCTETS

Source Port Number

Destination Port Number

UDP Length

UDP Checksum

Frame Format
DATA
(0-65508)

Within IP

IP HEADER

UDP HEADER

UDP DATA

Note that the UDP checksum covers both the UDP header and the UDP data fields.
This is distinct from the IP checksum which is a header checksum only, and does
not cover the integrity of the data.
This is not normally a problem with IP, as the data-link layer protocols often
provide a checksum for the network layer data. However, in some cases (e.g. SLIP)
such a check is not offered, and so it is required at this level.
However, if the receiving host detects an error in the UDP data frame, it will not
return an error to the sending host, but simply discard the frame - the same
behaviour that IP demonstrates.

UDP - Issues

UDP - Issues
Unreliability
No good for streamed services where reliability is
important, where a connection is needed

Examples
FTP
Mail
Telnet
WWW

TCP - Fundamentals

TCP - Fundamentals
Transmission Control Protocol (TCP)
Connection-oriented
Reliable

Used for
FTP
Mail
News
Web
And many more

TCP - Frame format

TCP - Frame format

2

Frame Format

OCTETS

SOURCE PORT NUMBER

DESTIN ATION PORT NUMBER

SEQUENCE NUMBER
ACKNOWLEDGEMEN T NUMBER

4 Bits

SEE

LENGTH RESERVED EXPANDED

TCP CHECKSUM

WINDOW SIZE
URGENT POINTER

OPTIONS (if any)

DATA
(0-????)

EXP ANDED
SECTION

U A
R C
G K

P
S
H

R
S
T

S
Y
N

F
I
N

(6 bits)

As for UDP there is still a source & destination port number.

The sequence number field identifies the byte in the data session stream that the
first byte of the frame represents. This is so that the frames can be reassembled in
the correct order at the other end. TCP (a connection-oriented) protocol is run over
IP (connectionless), and so this mechanism is needed for the destination host to
make sense of the order of the incoming frames.
The acknowledgement number field allows the receiving host to respond with an
acknowledge that it has correctly recieved the frame. This is the basis of TCPs
reliability. If a frame is not acknowledged, then it is send out again until it is.
A header length field is necessary as there are a number of TCP options (see
overleaf) that can be included.
The six flags in the expanded section involve various control messages that are sent
in order to set up the connection or to close it. They have the following meanings:
SYN - Synchronise sequence numbers to initiate a connection
ACK - Acknowledgement frame
RST - Reset the connection
FIN - Sender has finished sending data
URG - Urgent incoming data
PSH - Pass this data onto the application as soon as possible

TCP - A Sample Session

TCP - A Sample Session

3 Phases:

1
2
3

ESTABLISHMENT

Host
A

S YN <seq1> <ms s>

ACK <seq2>+1 <mss >, SYN <seq2 >

Host
B

ACK <seq2>+1

DATA TRANSFER

Host
A

D ATA x n
D ATA x m

Host
B

TERMINATION
FIN <seq1>+n, ACK <seq2>+m

Host
A

ACK <seq1>+n+1
FIN <seq2>+m, ACK <seq1>+n+1

Host
B

ACK <seq2>+m+1

mss stands for Maximum Segment Size, and is a limit on the max size of data
frames that can be sent to the host. This is primarily useful if the host in on a
network with a small MTU, and so wants to avoid too much fragmentation of the
data.
The reason we have 4 termination signals, but only 3 to set up the connection is that
each side of the connection must be shut down individually. TCP is full-duplex
communication, and half-duplex can be achieved by performing a half-close - i.e.
shutting only one side of the connection down.

TCP - Options

TCP - Options
End of
option
list

1
kind=0

1
No
operation
Max
segment
size
Window
scale
factor
Timestamp

kind=1

kind=2

len=4

max segment
size

kind=3

len=3

shift
count

kind=8

len=10

4
Timesta mp Value

4
Timestamp echo reply

Name Services

Name Services
Several Name Services exist
Common examples include
- DNS
- LDAP

System databases and Name Services are

configured in /etc/nsswitch.conf

Domain Name Service (DNS)

Domain Name Service (DNS)

DNS maps IP addresses to computer names and
vice versa
Mapping achieved with a name server

Provides a more user friendly address format

Hierarchical structure
DNS does other things too

Holds service records on where to find things

SRV records
Hold mail router information
MX records

Domain Name Service: Examples

Domain Name Service: Examples

Form of DNS Address
company.com

- A companys domain name

ahost.mycompany.com
- A computer in the domain

www.company.com

- A web server companys domain

www.othercompany.co.uk

- A web server for a UK company

Structure of DNS name space

Structure of DNS name space

.

com

org

uk

co

org

gov

The DNS system knows a hierarchical structure:

The root node(RED) is the "dot" domain. This dot is the origin of all domains. It is comparable
with the root of a UNIX filesystem.
Below the root node you will find a number of Top Level Domains (YELLOW). These can
further be distinguished in Generic Top Level Domains (gTLD), such as com, org and
net, and Country Code Top Level Domains (ccTLDs), such as nl (for the Netherlands),
au (for Australia) and uk (for the United Kingdom).
Below a Top Level Domain an organization can apply for a subdomain. The application
criteria and procedure for this varies from TLD to TLD. When an application has been
granted, then that organization becomes the "owner" of a domain, and can use it to
store information about its own hosts and (possibly) other subdomains.
Furthermore, the DNS system is decentralized. This means that there is no central
database which holds all the information, but organizations all keep their own databases on
their own servers. Through special so-called "glue records", these databases all point to
each other, making global lookups possible

LDAP

LDAP
Lightweight Directory Access Protocol
Protocol for enabling anyone to locate organizations,
individuals, and other resources such as files and
devices in a network
LDAP has a universal format, which supports display
of all names.
A typical LDAP server is a simple network-accessible
database where an organization stores information
about its authorized users and what privileges each
user has.
Provides centralised management

/etc/nsswitch.conf

/etc/nsswitch.conf
Configuration file for name services and system
databases
Tells system where to find user data and
computer name/IP address mapping
Data locations include
files

- e.g. /etc/passwd, /etc/shadow, /etc/hosts

dns
ldap

Connect to another host [telnet]

Connect to another host [telnet]

$ telnet hostname [port]
Connect to a remote host whose name is hostname
for the purposes of transferring running some
interactive programs
[port] by default this is 23 but could be others
e.g. 80 (http), 25 (SMTP), 143 (IMAP)
neo$ telnet trinity
Trying trinity.matrix.com...
Connected to 10.1.5.1.
Escape character is '^]'
SunOS 5.9
login:

Fetching remote files [ftp]

Fetching remote files [ftp]

$ ftp hostname
Connect to a remote host whose name is
hostname
for the purposes of transferring files
[port] by default this is 23 but could be
others e.g. 80
neo$ ftp trinity
Connected to 10.1.5.1.
220 trinity FTP server (Unix System V) ready
Name (10.1.1.1:unknown): morpheus
331 Password Required
Password: *****
230 User morpheus logged in.
ftp>

Inside ftp

Inside ftp
The following commands are available at the ftp>
prompt
list the files in the current remote directory
changes the current remote directory
displays the current remote director y
execute the command command locally
specifies the time of file to be transferred
gets the file filename from remote directory
and makes a local copy
put filename puts the file filename fr om local directory
and makes a local copy in the remote director y
mput|mget
allow multiple puts or gets with wildcards to be
done
quit
ends the ftp session

nlist|dir|ls
cd
pwd
!command
ascii|bin
get filename

Secure remote connections [ssh]

Secure remote connections [ssh]

$ ssh [-l login_name] [hostname |
user@hostname] [command]
Makes secure connection to hostname using public key
encryption
By default, connects as the current user
Can specify a command to run on the remote machine
[-l login_name] username to connect to host as
[user@hostname] alternative way to specify user
[hostname]
host to connect to
[command]
command to run on remote
machine
neo$ ssh trinity
root@trinitys password: *****
trinity$

Remote login without passwords [ssh]

Remote login without passwords [ssh]

SSH secures connections using public/private
keys
Public keys can be shared between machines to
allow non-interactive login
Useful to allow scripts to run things on remote
machines without needing to hardcode a password
Can be dangerous as if one machine is compromised,
many can be

Keys are created with ssh-keygen

Startup and shutdown

Startup and shutdown

There is only one satisfying

way to boot a computer
J. H. Goldfuss

Boot Stages

Boot Stages
There are three distinct stages in the boot flow.
Firmware
Kernel Initialisation & hardware
Software run levels

Briefly, it can be summarised as follows

POST

runs rc scripts

PROM loads bootstrap

runs init

loads k ernel

k ernel initialises

The power on self tests are used when a system is powered on or reset from the
PROM level. They check out the CPU and the MLB and the tests can vary from
system to system. By default the POST output is not displayed to the monitor,
however it can be viewed by connecting an ASCII terminal to the serial port.
User commands are used to control the operation of the PROM. They can change
PROM variables such as the boot device and security modes.
The PROM contains an entire programming language and operating system known
as forth. Developed in 1978, this high-level language can fit into an 8KByte chip.

Firmware

Firmware
Power on

Power On Self Tests

PROM loads boot block

banner

Test memory

The output from the POST can be seen on an

ASCII terminal attached to Serial Port A of the
machine

POST output

POST output
ok setenv diag-switch? true
ok setenv auto-boot? false
ok reset
PROM Checksum test
Context Reg Test
Setting Segment Map
Sizing Memory
Mapping ROM
Setting up RAM for monitor.
Setting up memory used in decompress.
Decompressing code to RAM ...
Remapping monitor's virtual addresses to RAM.
Probing /sbus@1,f8000000 at 0,0 dma esp sd st le
Probing /sbus@1,f8000000 at 1,0 Nothing there
Probing /sbus@1,f8000000 at 2,0 Nothing there
Probing /sbus@1,f8000000 at 3,0 bwtwo
Can't open input device.
SPARCstation IPC, No Keyboard
ROM Rev. 2.4, 12 MB memory installed, Serial 27831.
Ethernet address 8:0:20:a:e9:da, Host ID: 52006cb7.

The tests carried out by POST are simple yes/no type tests. It checks the integrity
of the CPU, hardware and any devices it can probe.

POST output
Testing
Testing audio chip; listen for a beep
Synchronous Error Reg Test
Synchronous Virtual Address Reg Test
Asynchronous Error Reg Test
Asynchronous Virtual Address Reg Test
System Enable Reg Test
Testing 12 megs of memory. Still to go
Cache Tag Memory Test
Cache Data Memory Test
SBus
SBus
SBus
SBus

slot
slot
slot
slot

0 le esp dma
1
2
3 bwtwo

Type b (boot), c (continue), or n (new command mode)

>

Open Boot PROM (OBP)

Open Boot PROM (OBP)

The boot PROM consists of

Power On Self Tests (POST)

Device drivers to know what to boot from
User commands (Forth Toolkit)
User diagnostics (Forth Toolkit)
User definable parameters

It is a single, upgradeable chip on the Main

Logic Board.
The current revision is OBP 4.x
The power on self tests are used when a system is powered on or reset from the
PROM level. They check out the CPU and the MLB and the tests can vary from
system to system. By default the POST output is not displayed to the monitor,
however it can be viewed by connecting an ASCII terminal to the serial port.
User commands are used to control the operation of the PROM. They can change
PROM variables such as the boot device and security modes.
The PROM contains an entire programming language and operating system known
as forth. Developed in 1978, this high-level language can fit into an 8KByte chip.

OBP: boot command

OBP: boot command

To start the OS from the OBP ok prompt use
boot [device] [options]

Boot options include

no options
[cdrom]
[net]
- [:dhcp]
- [ - install]
[-r]
[-s]
[-v]
[-a]

boot from default device

boot from cdrom
boot from networ k using A RP/RARP
use DHCP to fetch netw ork address
perform a Jumpstart installation
reconfigure boot
boot to r un level 1 (Single user)
verbose output
interactive boot

Kernel Initialisation

Kernel Initialisation
bootstrap file loads k ernel

check s/rebuilds devices

identifies root, swap & dump

k ernel identification
reports memory,
ethernet address,
CPU type.
executes init

The operating system is loaded from the boot

PROM
The kernel is the core of the operating system

The /etc/system file

The /etc/system file

/etc/system contains system parameters
Read by kernel on boot
The following types of customization are available in the
/etc/system file:

moddir
forceload
exclude
rootfs

rootdev
set

Changes path of kernel modules.

Forces loading of a kernel module.
Excludes a particular kernel module.
Specify the type for the r oot file system.
(ufs is the default.)
Specify the physical device path for root.
Set the value of a tuneable system parameter.

Incorrect editing of /etc/system can render a

machine unbootable!
If the /etc/system file is edited, it is strongly recommended that a copy of the
working file be made to a well-known location. In the event that the new
/etc/system file renders the system unbootable, it might be possible to bring the
system up with a boot -a command that specifies the old file. If this has not been
done, the system may need to be booted from CD or network so that the file can be
mounted and edited.

Boot output

Boot output
{ok} boot
Boot device: disk File and args:
Loading ufs-file-system package 1.4 04 Aug 1995 13:02:54.
FCode UFS Reader 1.12 00/07/17 15:48:16.
Loading: /platform/SUNW,Ultra-Enterprise/ufsboot
Loading: /platform/sun4u/ufsboot
SunOS Release 5.9 Version Generic_112233-01 64-bit
Copyright 1983-2002 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
configuring IPv4 interfaces: hme0.
starting DHCP on primary interface hme0
Hostname: trinity
The system is coming up. Please wait.
starting rpc services: rpcbind done.
Setting default IPv4 interface for multicast: add net 224.0/4: gateway
10.42.0.1
syslog service starting.
syslogd: line 24: WARNING: loghost could not be resolved
volume management starting.
The system is ready.
trinity console login:

init and run levels

init and run levels

init reads /etc/inittab

/sbin/rc3

executes /sbin/autopush

/sbin/rc2

/sbin/rcS

login prompt

Solaris works in one of a number of operational states

called run levels
A run level is a mode of operation in which certain
resources are made available
- a known set of processes are running
- certain files are mounted and (possibly) shared
- Solaris run levels are different to other UNIXes

Predefined run levels for Solaris

Predefined run levels for Solaris

Run
level

Action

Bring the sy stem to P ROM monitor lev el

Bring the sy stem to S ingle user lev el w here some file sy stems are
mounted and user log ins are disa bled.

M ultiuser lev el w ith no (NFS ) resources sha red.

F ull multiuser lev el w ith (NFS ) resources share d.

H alt the sy stem and boot interactiv ely (boot -a). or

H alt the sy stem and pow eroff (sun4m architecture)

H alt and reboot to run lev el 3.

S, s

Bring the sy stem to S ingle user lev el w here some file sy stems are
mounted and user log ins are disa bled.

Q, q

F orce the sy stem to re-read /etc/inittab.

a,b,c

O ptional and configurable ru n lev els.

Solaris run levels differ from other UNIX-like

systems
Other UNIX-like OSs have different definitions
e.g. Linux
Runlevel
System State
0
Halt System
1
Single user mode
2
Basic multi user mode without NFS (The same as 3, if you
don't have networking)
3
Full multi user mode (text based)
4
unused
5
Multi user mode with GUI
6
Reboot System

The /etc/inittab file

The /etc/inittab file

This file tells the init process what processes to
create for each run level.
# cat /etc/inittab
ap::sysinit:/sbin/autopush -f /etc/iu.ap
fs::sysinit:/sbin/rcS>/dev/console 2>&1 </dev/console
is:3:initdefault
p3:s1234:powerfail:/usr/sbin/shutdown -y -i5 -g0 >/dev/console 2>&1
. . . . . . . . . . . . . . . . . . . . . .

The /etc/inittab file defines three main items for the /sbin/init process.
The systems default run level.
What processes to start, monitor, and restart if they die.
What actions to be taken when the system changes run levels.

The /etc/inittab file

s3:3:wait:/sbin/rc3>/dev/console 2>&1 </dev/console

Each entry contains four fields separated by

colons.

id
s3
rstate
3
action
wait
command :/sbin/rc3>/dev/console

2>&1 </dev/console

id
This is a field of up to four characters which is used to uniquely identify the
entry.
rstate This field defines at which run level to execute.
action This key word field tells init how to treat the process.
Valid keywords are as follows:
initdefault
The default run level.
respawn
Restart the process when it dies.
powerfail
Start the process when init receives a power
fail. (Useful for UPS systems.)
sysinit Start the process and wait for completion.
wait Start process and wait for it to finish.
cmd The command or script to execute.

Run Control Scripts

Run Control Scripts

Run Control Script for each level calls further
script files for that level.
/etc

rc0

rc0.d

rc1

rc1.d

rc2

rc3

rc2.d

rc5

rc3.d

rc6

rcS.d

rcS

init.d

K20lp
S15nfs.server

For each run level there will be an entry in /etc/inittab which calls the rc scripts.
These controlling scripts are located in /sbin but are linked to files in /etc. Each run
level script in turns executes the files found under the /etc/rc#.d directory in
alphanumeric order.
Each run control scripts found in these sub directories is either in the format K## or
S##. The K## files are kill files and are used to stop processes when the system
enters this run level. The S## files are start files and are used to start processes.
There is also a directory /etc/init.d which is used for stopping and starting
individual services or processes without having to change run levels. Each of these
scripts is hard linked to the corresponding files in the /etc/rc*.d directories. These
files are in simple to remember format are used with either the parameter stop or
start.
/etc/init.d/lp stop
/etc/init.d/sendmail stop ; /etc/init.d/sendmail start

A Typical Run Control Script

A Typical Run Control Script

Run Control files are simple Bourne shell scripts.
trinity$ cat /etc/init.d/lp
#!/sbin/sh
#
# Copyright (c) 1997, 2001 by Sun Microsystems, Inc.
# All rights reserved.
#
#ident "@(#)lp 1.10
01/11/04 SMI"
case "$1" in
'start')
if [ -z "$_INIT_PREV_LEVEL" ]; then
set -- `/usr/bin/who -r`
_INIT_PREV_LEVEL="$9"
fi
[ $_INIT_PREV_LEVEL = 2 -o $_INIT_PREV_LEVEL = 3 ] && exit 0
PRINTERS=/etc/lp/printers
CONFIG=configuration
...

The above example is used to start and stop the print process. The file has two
other names: /etc/rc1.d/K20lp and /etc/rc2.d/S80lp. The K name ensure that the
process is killed when entering run level 1, and the S name ensures that it is started
when entering run level 2.
Note : To disable a start (or kill) file do not simply add on an extension. The
conventional way is to change the K or S prefix to a lower case k or s. This works
because the /sbin/rc scripts only search for K* and S*.

The /etc/init.d directory

The /etc/init.d directory

The /etc/init.d directory contains hard
links to the files in /etc/rc*.d
This allows processes to be stopped or started
without having to change run levels.
trinity$
trinity$
trinity$

/etc/init.d/lp stop
/etc/init.d/volmgt stop
/etc/init.d/lp start

The /etc/init.d directory is used by the system administrator to start and stop system
daemons (processes) by hand. This is sometimes necessary when processes go
wrong, but is also useful to test updated rc scripts. Note that system processes
should always be stopped in this way, rather than simply using the kill command,
because some system processes may also require other processes to be stopped
and/or files (especially lock files) to be cleaned up.
The scripts in this directory are hard links to their appropriate S scripts and K
scripts found in the other directories. They do not have the S or K prefix because
the administrator explicitly passes a start or stop parameter depending on whether
the script should start or stop something.
Administrators usually create new run control scripts in this directory, and then link
them into the appropriate /etc/rc?.d directory with a K or S name and corresponding
number. Thus the administrator may define the run-level in which the processes are
started and stopped, and the order in which this occurs.

Run level commands

Run level commands

Four commands exist to change run levels:

shutdown
halt
reboot
telinit

Run level commands: shutdown

Run level commands: shutdown

The /usr/sbin/shutdown command is
normally used to change to run level S or run
level 0
shutdown [-y] [-gseconds] [-irun_level]

trinity$ shutdown -g300 -i0

Shutdown started. Sat Nov 25 11:03:00 BST 1995
Broadcast Message from root (console) on viper Sat Nov 25 11:03:00..
THE SYSTEM IS BEING SHUT DOWN NOW ! ! !
Log off now or risk your files being damaged.
The system is coming down. Please wait.
System services are now being stopped.
Print services stopped.
Stopping the syslog service

By default shutdown will take the system to run level S (single user mode). The
shutdown command will broadcast a warning message to all users who are currently
logged onto the system that it is about to go down. It will not inform any users who
may be NFS mounting directories. Use the rwall command to inform them.
The options available with this command are as follows :y shutdown the system with no operator intervention.
g A grace period before shutdown occurs (in seconds).
i Specify the run level to change to. Choices are 0, 1, 2, 5 & 6.
When in single user mode typing ^D (control-D) will move the system back to the
default run level (run level 3).
Note : It is good practice to exit any GUI environment (eg OpenWindows or CDE)
and change directory to root before running shutdown.

Run level commands: halt

Run level commands: halt

The /usr/sbin/halt command brings the
system to run level 0 (PROM monitor level).

trinity$ halt
sync file systems [2] [2] [3] [3] [4]
ok

Using either of these commands will not broadcast any warning messages to the
users. The commands begin the shutdown process immediately.

Run level commands: reboot

Run level commands: reboot

The /usr/sbin/reboot command performs a
clean shutdown and brings the system back to
run level 3 (default).
reboot [bootoptions]

Options can be passed to the boot command

with reboot
-- -r will perform a reconfiguration boot on reboot

The reboot utility restarts the kernel. The kernel is loaded into memory by the
PROM monitor, which transfers control to the loaded kernel.
Although reboot can be run by the super-user at any time, shutdown(1M) is
normally used first to warn all users logged in of the impending loss of service.
The reboot utility performs a sync(1M) operation on the disks, and then a
multi-user reboot is initiated.

Run level commands: telinit

Run level commands: telinit

$ telinit [0123456abcQqSs]
The /usr/sbin/telinit command can be
used instead of the shutdown, halt and
reboot commands to change system run
levels.
# init 6
INIT: New run level 6
The system is coming down. Please wait.
System services are now being stopped.
Print services stopped.
Stopping the syslog service
Nov 23 15:12:21 viper syslogd: going down on signal 15
The system is down.
rebooting....
SPARCstation 10 (1 X 390Z50), Keyboard Present
ROM Rev 2.10..............

The init command can move the system to any of the run levels. It is somewhat
more direct that shutdown, however, giving no opportunity for users to logout.

Devices

Devices
There are a number of mechanical
devices which increase sexual
arousal, particularly in women.
Chief among these is the
Mercedes-Benz 380SL convertible.

P. J. O'Rourke

Local Devices

Local Devices
Local devices are those which are directly
connected to the machine
These can include
Hard Disk Drives
CDROM Drives
Tape Drives and Jukeboxes

Device Naming Conventions

Device Naming Conventions

Device names in the OS have four parts

Controller number
Target ID or SCSI ID
Drive number or LUN
Slice or partition

Examples include
c0t0d0s0
c0t6d0s0

the first disk drive in a machine

the internal CDROM drive

Links to these devices can be found in

/dev/dsk and /dev/rdsk
These conventions are used within the operating system

Device aliases [devalias]

Device aliases [devalias]

Devices are also named
in the boot PROM
The command devalias
can be used to show the
available devices

ok devalias
screen /sbus@1,f8000000/bwtwo@3,0
ttyb /zs@1,f1000000:b
ttya /zs@1,f1000000:a
keyboard! /zs@1,f0000000:forcemode
keyboard /zs@1,f0000000
disk /sbus/esp@0,800000/sd@3,0
net /sbus/le@0,c00000
cdroma /sbus/esp@0,800000/sd@6,0:a
cdrom /sbus/esp@0,800000/sd@6,0:c
tape /sbus/esp@0,800000/st@4,0
floppy /fd
tape0 /sbus/esp@0,800000/st@4,0
tape1 /sbus/esp@0,800000/st@5,0
disk3 /sbus/esp@0,800000/sd@0,0
disk2 /sbus/esp@0,800000/sd@2,0
disk1 /sbus/esp@0,800000/sd@1,0
disk0 /sbus/esp@0,800000/sd@3,0
scsi /sbus/esp@0,800000
ok

To reference devices in the OBP, use the strings shown in devalias

Partitions

Partitions
Logical divisions within each physical disk.
- A single range of contiguous blocks
- A physical dis k may have up to eight partitions

Partitions are called slices in Solaris, and are indicated

by the letter s when naming each partition device.
Partitions may be formatted in a variety of ways
- Unix UFS
- Veritas VxFS
- A paritition may also be left raw
- e.g for kernel swap space

Why Partition?

Why Partition?
Simplifies Administration
Backup individual partitions
OS updates may only affect specific partitions
Partitions can exist on external disks

Improves Data Security

Localises damage thus protecting other partitions
Disconnect sensitive partitions

Performance Enhancement
Reduces seek time
Localises fragmentation
Partitions can be configured to suit file sizes

Partition Tables

Partition Tables
The Partition table of a disk is held in a Volume
Table of Contents (VTOC)
Partitions must be contiguous and must start
and end on cylinder boundaries
A Partition is defined as an offset and a size

O ffset

P artition 0

P artition 1

P artition 6

P artition 7

Size
55

Size
74

Size
167

Size
105

55

129

296

Initial partitioning is carried out at installation time and ensures there are no
overlapping or non-contiguous partitions. So, the first partition starts on cylinder 0
and continues through to the end of cylinder 54. The next partition then starts on
cylinder 55 and so on.
Should partitions need to be changed after installation time then the format utility is
available in Unix for this.
The output from prtvtoc reads the disk label and returns the partitioning
information.

Listing partitions [prtvtoc]

Listing partitions [prtvtoc]

/usr/sbin/prtvtoc [-fhs] device

The prtvtoc command is used to report disk

geometry and list partitions

[-f]
[-h]
[-s]
device

report disk free space

omit headers from report
only show column headers
name of the device to list

NAME
prtvtoc - report information about a disk geometry and partitioning
SYNOPSIS
prtvtoc [-fhs] [-t vfstab] [-m mnttab] device
DESCRIPTION
The prtvtoc command allows the contents of the VTOC (volume
table of contents) to be viewed. The command can be used
only by the super-user.
The device name can be the file name of a raw device in the
form of /dev/rdsk/c?t?d?s2 or can be the file name of a
block device in the form of /dev/dsk/c?t?d?s2.

prtvtoc output

prtvtoc output
trinity$
trinity$ prtvtoc
prtvtoc /dev/rdsk/c0t1d0s2
/dev/rdsk/c0t1d0s2
** /dev/rdsk/c0t1d0s2
/dev/rdsk/c0t1d0s2 partition
partition map
map
**
** Dimensions:
Dimensions:
**
512
512 bytes/sector
bytes/sector
**
80
80 sectors/track
sectors/track
**
19
tracks/cylinder
19 tracks/cylinder
**
1520
sectors/cylinder
1520 sectors/cylinder
**
3500
3500 cylinders
cylinders
**
2733
2733 accessible
accessible cylinders
cylinders
**
** Flags:
Flags:
** 1:
1: unmountable
unmountable
** 10:
10: read-only
read-only
**
First
Sector
Last
First
Sector
Last
** Partition
Sector
Count
Sector
Partition Tag
Tag Flags
Flags
Sector
Count
Sector Mount
Mount Dir
Directory
ectory
00
22
00
0
62320
62319
00
0
62320
62319
11
33
01
62320
197600
259919
01
62320
197600
259919
22
55
01
00 4154160
01
4154160 4154159
4154159
66
44
00
259920
3894240
4154159
00
259920
3894240
4154159

Creating partitions [format]

Creating partitions [format]

The format utility is used to define disk
partitions.
trinity$ format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t1d0 <CDC Wren IV 94171-344 cyl 1545 alt 2 hd9 sec 46>
/sbus@1,f8000000/esp@0,8000000/sd@1,0
1. c0t3d0 <SUN0207 cyl 1254 alt 2 hd 9 sec 36>
/sbus@1,f8000000/esp@0,8000000/sd@3,0
Specify disk (enter its number): 1

The format utility can only be run with root privileges and is a suite of programs
which allow the administrator to carry out a range of functions on the disk. When
format is run it will identify all disks known to the system. If the new disk is not
listed by format then there is no entry in /dev and /devices. A reconfiguration boot
will be required.

The format Main Menu

The format Main Menu

The format main menu offers a range of
functions.
FORMAT MENU:
disk
type
partition
current
format
repair
label
analyze
defect
backup
verify
save
inquiry
volname
quit
format> partition

select a disk
select (define) a disk type
select (define) a partition table
describe the current disk
format and analyze the disk
repair a defective sector
write label to the disk
surface analysis
defect list management
search for backup labels
read and display labels
save new disk/partition definitions
show vendor, product and revision
set 8-character volume name

Menu choices can be abbreviated to two letters. The partition option is used to
partition the disk selected in in the previous menu.

The partition Menu

The partition Menu

This menu allows the modification of individual
partitions or the entire disk
PARTITION MENU:
0
1
2
3
4
5
6
7
select
modify
name
print
label
quit
format>

change '0' partition

change '1' partition
change '2' partition
change '3' partition
change '4' partition
change '5' partition
change '6' partition
change '7' partition
select a predefined table
modify a predefined table
name the current table
display the current table
write partition map and label to the disk

The partition menu offers the following functions :0-7

Specify the Tag, Flag, Offset and Size of the partition.
select Select a pre-defined partition table.
modify Modify a pre-defined partition table.
name Name the current partition.
print Display the current partition table.
label Write the current table to the disk VTOC.

Partitioning a disk using modify

Partitioning a disk using modify

partition> modifya new partition table.
Preparing

Select partitioning base:

0. Current partition table (original)
1. All Free Hog
Choose base (enter number) [0]? 1
Part
Tag
Flag
Cylinders
Size
0
root
wm
0
0
1
swap
wu
0
0
2
backup
wu
0 - 1253
198.39MB
3 unassigned wm
0
0
4 unassigned wm
0
0
5 unassigned wm
0
0
6
usr
wm
0
0
7 unassigned wm
0
0

Blocks
(0/0/0)
(0/0/0)
(1254/0/0)
(0/0/0)
(0/0/0)
(0/0/0)
(0/0/0)
(0/0/0)

0
0
406296
0
0
0
0
0

Do you wish to continue creating a new partition

table based on the above table[yes]? yes
Free Hog partition[6]? 7

The modify option can only be used on an unmounted disk. It allows you to specify
the size of each partition on the disk (except partitions 2 and the Free Hog partition)
but it will not allow you to change either the Tag or the Flag.
The Free Hog partition is used to store all the disk space and this contracts as space
is allocated to each partition.
In the above example we will be creating three partitions as follows :Partition 0
50 Mbytes
Partition 1
125 Mbytes
Partition 7
The rest (Free Hog)

Partitioning a disk using modify

Enter size of partition
[0b, 0c,
0.00mb]:
50mb
Dividing
the disk'0'into
three
partitions
.
Enter
Enter
Enter
Enter
Enter
Part
0
1
2
3
4
5
6
7

size
size
size
size
size

of
of
of
of
of

partition
partition
partition
partition
partition

Tag
root
swap
backup
unassigned
unassigned
unassigned
usr
unassigned

Flag
wm
wu
wu
wm
wm
wm
wm
wm

'1'
'3'
'4'
'5'
'6'

[0b,
[0b,
[0b,
[0b,
[0b,

0c,
0c,
0c,
0c,
0c,

Cylinders
0 - 316
317 - 1107
0 - 1253
0
0
0
0
1108 - 1253

0.00mb]:
0.00mb]:
0.00mb]:
0.00mb]:
0.00mb]:

125mb
<return>
<return>
<return>
<return>

Size
50.15MB
125.14MB
198.39MB
0
0
0
0
23.10MB

Blocks
(317/0/0)
(791/0/0)
(1254/0/0)
(0/0/0)
(0/0/0)
(0/0/0)
(0/0/0)
(146/0/0)

102708
256284
406296
0
0
0
0
47304

Okay to make this the current partition table[yes]? <return>

Enter table name (remember quotes): "Big Al"
Ready to label disk, continue? yes
partition>

The modify option takes care of sizes and offsets and ensures there are no
overlapping or unused space. Sizes can be specified in either Blocks, Cylinders or
Mbytes and modify will always round up or down to the nearest cylinder boundary.
The table name is optional and can be up to eight characters in length. If you want
to include spaces then the name must be in quotes.
The label option at the end of modify writes the new partition table back to the disk.

File Systems

File Systems
File systems are ways of organising the way
data is stored on disks
The default file system in Solaris is ufs
Another popular file system is Veritas journaling
VxFS
Journaling file systems often provide better resilience
and faster recovery than non-journaling file systems
Journals should be stored somewhere sensible
- Journal in swap may be fast, but does not survive a
reboot, thus useless.

Network File Systems

Network File Systems

Network file systems allow access to storage not
physically attached to the machine
They use the network to access the data
Types of network file systems include
NFS
CIFS
SMB

While useful, network file systems can cause

problems
NFS timeouts on remotely mounted drives can lead to
machines hanging

Media errors

Media errors
Errors occur on all media
Disk, tape, CDROM
Disk errors are reported to the console window
Error messages are also logged in the file
/var/adm/messages
Error for command 'read(10)' Error Level: Retryable
Requested Block 2422272, Error Block: 2422283
Sense Key: Media Error
Vendor 'SEAGATE':
ASC = 0x11 (unrecovered read error), ASCQ = 0x0, FRU = 0x0
Error for command 'read(10)' Error Level: Retryable
Requested Block 2422272, Error Block: 2422283
Sense Key: Media Error
Vendor 'SEAGATE':
ASC = 0x11 (unrecovered read error), ASCQ = 0x0, FRU = 0x0

Checking and fixing file systems [fsck]

Checking and fixing file systems [fsck]

The fsck command is used to check the
integrity of file systems
It can also be used to interactively repair
inconsistencies

fsck [-F FSType] [-m] [-V] [special...]

fsck [-F FSType] [-n | N | y | Y]
[-V]
[-o FSType-specific-options] [special...]

NAME
fsck - check and repair file systems
SYNOPSIS
fsck [-F FSType] [-m] [-V] [special...]
fsck [-F FSType] [-n | N | y | Y] [-V] [-o FSTypespecific-options] [special...]
DESCRIPTION
fsck audits and interactively repairs inconsistent file system conditions. If the file system is inconsistent the
default action for each correction is to wait for the user
to respond yes or no. If the user does not have write permission fsck defaults to a no action. Some corrective
actions will result in loss of data. The amount and severity
of data loss can be determined from the diagnostic output.

Backup: Principles

Backup: Principles
Backups must be rigorously performed
Follow a well defined strategy

Restore backups regularly

Check that backups are being written to the media
correctly

Implement a disaster recovery plan

Test out a disaster recovery (if possible)
Well documented, preferably with an off-site copy

Backup: Principles
Store backup media properly
Keep away from any electro-magnetic fields
Store off-site if possible

Prevent accidental erasure of a backup

Enable write-protect on backups
Label backups clearly
Well documented rotation strategy

Clean backup systems

Keep tape heads clean

Backup Commands: tar

Backup Commands: tar

tar is a universal Unix command, usually used to
back up files and directories
tar [crtux] [bflLmovw] filename

[-c]
[-r]
[-t]
[-u]
[-x]
[-f filename]

create a tar archive

append files to archive
list files in archive
add files not in archive
extract files from archive
send to device at filename or
create tar file called filename

The tar command is commonly used to backup files and directories. It can be used
by any user and is relatively simple in its syntax. In general tar is very portable
between different platforms and Operating Systems but has the drawback of not
being able to handle multiple-volumes.
When using tar , files and directories are all compacted into one single file (a tar
file) and this is specified with the -f filename option. Normally this is a device,
such as /dev/rmt/0, but can also be a file. This is very useful if you wish to email a
number of files and directories to another person. By creating a single tar file it is
much easier to send.

Backup Commands: tar examples

Backup Commands: tar examples

To create a tar archive.
trinity$ cd /export/home/morpheus
trinity$ tar -cvf /dev/rmt/0 .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .

To extract a single file or the whole backup.

trinity$ tar -xvf
. . . . . . . . .
trinity$ tar -xvf
. . . . . . . . .
. . . . . . . . .

/dev/rmt/0 ./games
. . . . . . . .
examples.tar
. . . . . . . .
. . . . . . . .

To list the contents of the archive.

trinity$ tar -tvf /dev/rmt/0
. . . . . . . . . . . . . .

Take care when using tar to always specify the relative pathname of the files and
directories to be archived. This will ensure that the archive is portable and may be
restored.
By using an absolute pathname when writing the archive, the files can only be
extracted into exactly the same directory path. This tends to limit the portability of
the files being archived.

Backup Commands: cpio

Backup Commands: cpio

cpio is commonly used in command pipelines.
cpio supports multi-volume backups.
cpio -[iot] [pvdmBH]

[-i]
[-o]
[-t]

input (read)
output (create)
list

Although not as syntactically straightforward as tar, cpio is a powerful command

for storage and retrieval. There are many additional options for cpio which are
listed in the man pages.
cpios major distinction from tar is that it allows the files to be archived to be
specified on the standard input. Consequently, cpio especially useful when used in
conjunction with commands that generate file names. For example, it is common
for cpio to be used in conjunction with find to selectively backup files. In the
following example, all files owned by fred are archived.
find . -user fred -print | cpio -o > /dev/rmt/0

Backup Commands: cpio examples

Backup Commands: cpio examples

To create a cpio backup.
trinity$ find . -print | cpio -ovB > /dev/rmt/0
. . . . . . . . . . . . . . . . . . .

To restore from a cpio backup.

trinity$ cpio
. . . . . . .
trinity$ cpio
. . . . . . .

-ivdmB < /dev/rmt/0

. . . . . . . . . . .
-ivdmB fil* < /dev/rmt/0
. . . . . . . . . . .

To list the contents of the backup

trinity$ cpio -itvB < /dev/rmt/0
. . . . . . . . . . . . . . . . . .

Backup Commands: dd

Backup Commands: dd
dd provides access to raw devices
dd [option=value]

It may be used to tar to and from a remote

machine ...
trinity$ tar cvf - . | rsh smith dd
. . . . . . . . . . . . . . . . . .
trinity$ rsh smith dd if=/dev/rmt/0
. . . . . . . . . . . . . . . . . .

of=/dev/rmt/0
. . . . . .
| tar xvBpf . . . . . .

... or to convert file formats

trinity$ dd if=/dev/rmt/0 conv=swab | tar xvf -

The dd command stands for copy and convert and is literally a bit-copier. For this
reason it is useful when converting data between raw physical devices, and for
copying the entire contents of one disk to another disk or for duplicating tapes.
It is not usually used for backup purposes as it is slow and provides little or no
verbose output. However, when combined with utilities such as tar, it supports
remote backups.
In the final example above dd is used to convert the byte ordering of the data on
tape, before passing this data to tar for extraction. Such a command would be
useful when reading tapes on a Sun system which had been written on a byte
swapped machine such as those produced by Silicon Graphics.
The copy-and-convert utility is called dd rather than cc, because when it was
written for Unix cc had already been claimed by the C compiler. dd was the
next best thing!
dd is particularly useful for making backups of CDs to ISO9660 image files.

Backup Commands: mt

Backup Commands: mt
mt provides controls for magnetic tape drives
mt [-f tape_name] command [count]

fsf
bsf
rewind
retention
erase
#
#
#

forward space over count EOF marks

backward space over count EOF marks
rewind the tape
rewind and wind the tape forward
erase the entire tape

mt -f /dev/rmt/0 rewind
mt -f /dev/rmt/0n fsf 2 ; ufsrestore -rvf /dev/rmt/0n

The /usr/bin/mt command is used to position and control a magnetic tape. The mt
command is useful when several archives (for example, ufsdump, tar, cpio) must
share the same tape. mt allows early archives to be scanned past, so that subsequent
backups appear later on the tape.
Note that in order for several backups to share a tape it is important that the device
is identified by its no - rewind name. /dev/rmt/0n tells the device driver
controlling access to the tape drive not to rewind when the device file is closed.