Вы находитесь на странице: 1из 97

Operating Systems

The structure of a Computer system can be visualized as having four basic components:
1. Hardware – Provides basic computing resources - CPU, memory, I/O devices
2. Operating system – Controls and coordinates use of hardware among various applications and
users
3. Application programs – Define the ways in which the system resources are used to solve the
computing problems of the users
4. Users – Anybody who desires work to be done by a computer system. People, machines, other
computers

Types of OS

OS are classified into different types depending on their capability of processing.

Single User and Single Tasking OS:

These are simple operating system designed to manage one task at a time ,for use by a single
user for a standalone single computer for performing a single task .

Single User and Multitasking OS:

These OS allow execution of more than one task or process concurrently by dividing the
processor time amongst different tasks.

Multi-programming OS:

These OS allow more than one programs to run at the same time .
Real Time OS:

These are designed to respond to an event within a predetermined time. These operating
systems are used to control processes

Embedded OS:

Embedded in a device in the ROM. They are specific to a device and are less resource intensive.

Function Of OS

The OS performs basic tasks such as controlling and allocating memory, prioritizing system
requests, controlling input and output devices, facilitating networking, and managing files

UNIX Introduction

 Unix is an OS for Programmers as shell(the command interpreter)provides the


programming facility.
 It provides an in-built security mechanism through the user name and password,
combined with the access rights associated with files
 Developed by Ken Thompson and Ritchie originally in assembly, and later in C, thus
making it portable to other machines

Supports C, Fortran, Basic, Pascal, COBOL, Lisp, Prolog, Java, Ada compilers

Features of Unix Operating System

• Multi-user and Multitasking


• Everything is a file
• Configuration data stored in text
• Small, single purpose programs
• Ability to chain programs together to perform complex task
• Facility of background processing

Architecture of the UNIX System

The UNIX operating system has a layered architecture having three main components
• Kernel

• Shell

• Utilities
The Unix Kernel

Kernel is a collection of programs mostly written in C which allocate the system resources and
coordinate all the details of the computer's internals.

Functions of Kernel:

 It allocates time and memory to programs and handles the file store and
communications
 Interacts directly with the hardware through device drivers
 Provides sets of services to programs
 Manages memory, controls access, maintains file system, handles interrupts, allocates
resources of the computer

System calls

The system calls are functions used in the kernel itself. UNIX system calls are used to manage the
file system, control processes, and to provide interpocess communication.

System calls can be categorized as:

 File structure related calls -For example create, open,read, write,lseek,dup etc.
 Process Related calls -For example fork,exec,wait,exit etc
 Inter process related calls - For example pipe,msgget,msgsnd etc.

The Unix File System:

It is a hierarchical collection of 3 types of files:

 ordinary
 directory
 special files (device, pipe, fifo, socket).

A UNIX file is featureless because it is simply an array of bytes.


Dominant file type in UNIX is the text file.
System related files are also stored in text form.
Separate device can be added by creating a file for it.

Root is the supremo and is represented by the ‘/’. Every subdirectory must have a parent.

File names can be up to 14 characters long; can contain both upper and lower case alphabets,
digits, a dot, hyphen (-), underscore (_) anywhere; should not have a blank or tab; are case-
sensitive.

Path names are a sequence of directory names separated by ‘/’. They are used to access files.

Absolute pathname - file location is determined with respect to the root.

Relative pathname - file location is determined with respect to the current directory.

Though the UFS looks hierarchical, it is actually a directed acyclic graph because files can be
shared.
/home – It holds user’s home directories. In other UNIX systems, this can be /usr directory.

/bin – It holds many of the basic Linux programs; bin stands for binaries, files that are executable.

/usr – It holds many user-oriented directories:

/sbin – It holds system files that are usually run automatically.

/etc – It and its subdirectories hold many of Linux config files.

/dev – It holds device files. All info sent to /dev/null is thrown into trash. Your terminal is one of
the /dev/tty files.

The UFS resides on a single logical disk. A logical disk is a disk partition comprising of a set of
consecutive cylinders.

UFS further subdivides a partition into one or more cylinder groups and attempts to allocate
inodes and related data blocks from the same cylinder group, thus minimizing the disk head
movements.

At the beginning of the logical disk lies the boot block of UNIX operating system containing the
bootstrap program.

It is followed by repetitive cylinder groups each one containing a super block, cylinder group
block, inode list and the data area.
Each cylinder group contains a duplicate copy of the super block. The super block contains the
size of file system, number of free blocks, index of next free block in free block list, size of inode
list, number of free inodes, index of next free inode in free inode list.

The cylinder group block contains a number of inodes and corresponding data blocks for that
cylinder group. The block size is a power of 2 (>=4096).

Internal File Maintenance

For each file created in the system, an inode is also created. Inode is a disk file record of 64 bytes
that maintains the permanent attributes of a file.

An inode is permanent and it exists until the corresponding file is removed from the system.
Sample details of an inode –

 Owner and group identifiers


 File type and file size
 Number of links for this file
 Times of file creation, last file access and modification, and last inode modification
 List of access rights – read/write/execute permissions
 Reference count showing number of times file is opened
 Physical address of file on the disk: array of 13 pointers for data storage

Whenever a file is opened, its inode is brought into main memory. The active inode is kept there
until the file is closed and is used to locate the beginning of an open file on disk and to verify that
every I/O request comes from a valid user as per specified access permissions.
Users and Access Rights

Objective
This module is to understand about different user types in UNIX and their access rights.

Course Content

 User types
 File access permissions
 Changing file access permissions using symbolic mode
 Changing file access permissions using octal mode

User Types in Unix

There are three types of accounts on a Unix system:


- Root Account
- System Account
- User Account

Managing users and groups


Unix supports a concept of Group which logically groups a number of accounts.
➢ Every account would be a part of any group.
➢ Unix groups plays important role in handling file permissions and process management.
➢ Grouping of users allows to grant and revoke file permissions collectively.

File Permission: Ownerships level


File ownership is an important component of Unix that provides a secure method for storing
files. Every file in Unix has the following attributes:
File Permission Types (Mode)
Three type of permissions can be set for any ownership level:
Read, Write and Execute Three modes or permissions have different meaning for file and
directory;

Changing Permissions (chmod)


To change file or directory permissions, the chmod (change mode) command is used. There are
two ways to use chmod:

 Symbolic mode
 Absolute mode
chmod command options:

Using chmod in Symbolic Mode:

With symbolic representation, permission set can be added, deleted, or specified using the
operators listed in the following table:

Example:

Let us assume there is a file name "abc" , whose initial permissions are only read permission to
user and group and others. We will provide full permissions to all three types of users.

Let us verify the existing file permissions using ls -l command as below and observe the next
steps, like changing the permissions and verifying the changes.
In the above example,

chmod u+wx,g+wx abc , adds write and execute permission to user and group to the file abc

chmod g-x abc, removes execute permission from group from the file abc.

chmod o=r-- abc, provides read permission and removes write and execute permission from
others for file abc

Using chmod in Absolute(octal) Mode

The second way to modify permissions with the chmod command is to use a number to specify
each set of permissions for the file. Each permission is assigned a value, as the following table
shows, and the total of each set of permissions provides a number for that set.
Examples: Observe the following sequence of command execution statements and analyze the
output

1. Let us view the initial permissions of the file abc


ls -l abc

-rw-r--r-- 1 735873 oinstall 0 Feb 7 12:37 abc

If we observe the output, initially owner have read and write permissions and group and others
have only read permission

2. Let us remove all the permission from all users for abc file, as below
chmod 0 abc
Let us verify the changed permissions

ls -l abc
---------- 1 735873 oinstall 0 Feb 7 12:37 abc
Yes all permissions are taken away from all users on the file abc

3. To provide all permissions to the user, read permission to group and write and execute
permissions for Others,

use the below command


chmod 743 abc

Let us verify the changed permissions


ls -l abc
-rwxr---wx 1 735873 oinstall 0 Feb 7 12:37 abc
From the output all permissions to user, read permission to group and write and execute
permissions for Others, Use the below command

4. chmod 755 abc


The above command provides all permissions to the user, read and execute permissions to
group and others
Let us verify ls -l abc
-rwxr-xr-x 1 735873 oinstall 0 Feb 7 12:37 abc

5. chmod 777 abc


The above command provides all permissions to all users . We can verify using below command
ls -l abc
-rwxrwxrwx 1 735873 oinstall 0 Feb 7 12:37 abc

Basic UNIX commands :


The basic Unix commands are:

echo

It displays a line text.

Echo the given string or strings to standard output.

Syntax:
echo [OPTION]... [STRING]...

Example:

echo "Good Morning"

This will display the following in the standard output

Good Morning

passwd command

The passwd command is used to change passwords for user accounts. A normal user may only
change the password for his or her own account, while the superuser may change the password
for any account.

Syntax : passwd [options] [LOGIN]

Options:
uname command

Print certain system information.

Syntax : uname [OPTION]...

Options:
Examples: To show the name of the OS

uname command:

Used to get the information about the current system (the name, version and other details)

$uname

To get the kernel name, you can use -s parameter.

$ uname -s

To get the kernel release, -r option is used

$ uname -r

who command:

who command prints information about all users who are currently logged in.
Syntax:
who [OPTION]... [ FILE ] [ am i ]
find command:

find is one of the powerful utility of Unix (or Linux). It can search
the entire file-system to locate files and directories according to the specific
search criteria. Other than searching files in can also perform actions(executing
commands) on searched files.
Syntax:

$ find [option] [action]

Examples :
1 . File Type based Search:
>> The following command finds the file named Abc in current(.) directory and
all its sub-directories

[390119@InGhyUnix ~]$ find . -name Abc


./Abc
./Dir1/Abc
2. Wild-Card based Search : Wild-card characters like * and ? can be used:
>> To find all files where filename starts with Abc the command will be:

[390119@InGhyUnix ~]$ find . -name "Abc*"


./Abc
./Dir1/Abc
./Dir1/Abc.php
./Abc.txt
./Abc.dat

3 . Source Location based Search:


>>The command find can search in multiple source location :

[390119@InGhyUnix ~]$ find Dir1 Dir2 -name "Abc.*"


Dir1/Abc.php
Dir2/Abc.htm

Searches for the file Abc with any extension only in directories Dir1 and Dir2

4 . Size Based Search : Using find files can be searched based on its size .
>> To find the files with size more than 10Mb the command will be

[390119@InGhyUnix ~]$ find . -size +10M

./Abc.dat

5 . Access/Modification-Time Based Search : Using find files can be searched


based on the time of last access/modification :
>> To find the files which are accessed within last 3 days the command will be

[390119@InGhyUnix ~]$ find . -atime -3

./Abc

6 . Permission Based Search : Using find command files can be searched


based on the access permission they have
>> To find the files having read write and execute permission only for the owner
the command will be :

[390119@InGhyUnix ~]$ find . -perm 700


./Abc.txt

7 . Perform Action on the search result : Using exec option any action (command)
can be executed on the search result.
>> To remove all the files having extension .swp from the current directory the
command will be :

[390119@InGhyUnix ~]$ find . -name “*.swp” -exec rm {} \;

bc command:

bc is the command used for basic mathematical calculations.

It can be used either a mathematical scripting language or as an interactive mathematical shell .

In interactive mode it accepts input from the terminal and providing calculations on requested
arithmatic expression.

That means typing the command bc on a Unix command prompt and entering a mathematical
expression, such as (1 + 4) * 2 , whereupon 10 will be the output

While used with files , bc starts by processing code from all the files listed on the command line
in the order listed. After all files have been processed, bc reads from the standard input. All code
is executed as it is read.

Syntax: bc [ -hlwsqv ] [long-options] [ file ... ]

Example:

$ echo "2+5" | bc

Output will be:

tty command:
Prints the file name of the terminal connected in standard input.
Syntax : tty [OPTION]...
UNIX Text Editors

To write a program or a paper even to edit a mail message, you need to create, write,store,
retrieve, view, and edit files. In unix several text editors available for this .

Editors available unix are ed, node,NEdit, emacs and vi .

One of the first end-user programs hosted on the system and standardized in UNIX-based
systems ever since is the vi editor. ed is a line editor for the UNIX operating system.

vi Editor:

vi (pronounced "vee-eye") is short for "vi"sual editor. It is a visual editor used to create and edit
text files.

It displays a window into the file being edited that shows 24 lines of text. vi lets you add, change
and delete text, but does not provide formatting capabilities like centering lines or indenting
paragraphs.

vi works in three different modes:

1. Edit Mode -where any key is entered as text


2. Command Mode -where keys are used as commands
3. Ex Mode -ex commands can be entered in last line to act on text

Invoking vi with/without filename puts it in command mode:

vi []

Commands to be used in vi:

For cursor movement within the vi editor, we can not use the mouse . We have to use key
commands. To use a key command, user has to be in the command mode. To go to command
mode, press the Esc key.
Commands to Enter and Exit vi:

command Purpose of the command

vi filename edit filename starting at line 1

vi –r filename recover filename that was being edited when system crashed

The new or modified file is saved when user exits from vi. But it is also possible to exit without
saving the file.

To use a command to exit from vi , after pressing the Esc key ,colon : is to be pressed.

The cursor moves to bottom of screen whenever a colon (:) is typed. This type of command is
completed by hitting the (or ) key.

command Purpose of the command

:wq quit vi, writing out modified file to file named in original invocation

:q quit (or exit) vi

:q! quit vi even though latest changes have not been saved for this vi call

:w write current contents to file named in original vi call

Commands to insert or add text:

Following are the commands to be used to insert or add text.

key Purpose of the command

u Undo the last command’s action

i
insert text before the cursor,until is hit
I
insert text at beginning of current line, until hit

a append text after cursor, until hit

A append text to end of current line, until hit

o open and put text in a new line below current line,


until hit

O open and put text in a new line above current line,


until hit

Commands to modify texts:

key Purpose of the command

r replace single character under cursor (no needed)

R replace characters, starting with current cursor position, until hit

cw change the current word with new text,


starting with the character under cursor, until hit

cNw change N words beginning with character under cursor, until hit;
e.g., c5w changes 5 words

C change (replace) the characters in the current line, until hit

cc change (replace) the entire current line, stopping when is hit

Ncc change (replace) the next N lines, starting with the current line,
or stopping when is hit
cNc

Commands to Delete texts:


key Purpose of the command

x delete single character under cursor

Nx delete N characters, starting with character under cursor

cw delete the single word beginning with character under cursor

dw change N words beginning with character under cursor, until hit;


e.g., c5w changes 5 words

dNw delete N words beginning with character under cursor;


e.g., d5w deletes 5 words

D delete the remainder of the line, starting with current cursor position

dd delete entire current line

Ndd delete N lines, beginning with the current line;


or e.g., 5dd deletes 5 lines
dNd

Commands to copy and paste text:

yy copy (yank, cut) the current line into the buffer

Nyy copy (yank, cut) the next N lines, including the current line, into the buffer

p put (paste) the line(s) in the buffer into the text after the current line

Commands for searching text

/string search forward for occurrence of string in text

?string search backward for occurrence of string in text

UNIX FILE COMMANDS

In this module following important file handling commands will be discussed:


 touch
 cat
 cat > (single redirection)
 cat >>(double redirection)
 cp
 mv
 cmp
 comm
 diff

touch command
Touch command is used in Unix to create an empty file.

$ touch file1 file2


$ ls -l
-rw-r--r-- 1 user group 0 2014-02-08 7:14 file1
-rw-r--r-- 1 user group 0 2014-02-08 7:14 file2
$

Touch command is used to change the timestamps (access time and modification time of a file).

cat command
Use of cat command:
'cat' command is used to display a file content.
Syntax:

$ cat filename [ filename ]

$ cat data
This is an unix file
This is line two
$

cat > and cat >> command


Redirection operator ( > and >>) can be used with cat command to take input from standard
input device(keyboard) and store them in a file.
Syntax: $ cat > [filename]

$ cat > data


The file created using
cat > filename
^d
$
cat with single redirection(cat >filename) , creates a new file if no file with the given file name
already exists and overrites the file if a file with given file name already exists.

cat with double redirection (cat >>filename) appends the content given through standard input
to the file.

cp command

cp command is used to copy one file's content to another file.


Syntax: $ cp [source filename] [destination filename]
Switches:
1. cp -i [sourcefilename] [destination filename] This command copies the content of a file
interactively.

2.cp -r [source_directory] [destination_directory] This command copies the whole file hierarchy.

3. cp -v [sourcefilename] [destination filename] Copies in verbose mode, explains what is being


done

mv command
mv command is used for:
1. for renaming a file in same directory.
2. for moving a file to a different directory.
Syntax:
$ mv [sourcefilename] [destinationfilename]
$ mv [source_directory] [destination_directory]

Directory Commands:

Every item in the UNIX filesystem tree is either a file, or a directory. A directory is like a file
folder. A directory can contain files, and other directories. A directory contained within another
is called the child of the other. A directory in the filesystem tree may have many children, but it
can only have one parent. A file can hold information, but cannot contain other files, or
directories.

The place in the file system tree where an user is located is called the current
working directory (which can be listed using pwd command).

As a user logs in, user is put in his/her home directory.

mkdir(make directory) command:


The mkdir command creates a directory with specified name in the present working directory or
specified path.
Syntax: $ mkdir [ ]
ls Command
Lists the contents of any directory (current directory if no directory specified).
Sorts entries without option
$ ls [option(s)] [filename]

File comparison commands:

cmp – This command is used to compare two files, and if they differ, tells the first byte and line
number where they differ. If input files are same , returns nothing.

Syntax :

cmp options file1 file2

To use cmp commands both the files need to be sorted.

comm- This command is used to compare two sorted files.

Syntax:

comm [options] file1 file2

 One set of options allows selection of ‘columns’ to suppress.


o -1: suppress lines unique to file1 (column 1)
o -2: suppress lines unique to file2 (column 2)
o -3: suppress lines common to file1 and file2 (column3)

Example: To show only lines common between file1 and file2, command is

$ comm -12 file1 file2

diff - This command is used to compare two files line by line. The output indicates how the lines
in each file are different, and the steps that will be involved in changing file1 to file2.

The change commands are in the format [range][acd][range]. The range on the left may be a
line number or a comma-separated range of line numbers referring to file1, and the range on the
right similarly refers to file2. The character in the middle indicates the action i.e. add, change or
delete.

 ‘LaR’ – Add lines in range ‘R’ from file2 after line ‘L’ in file1.
 ‘FcT’ – Change lines in range ‘F’ of file1 to lines in range ‘T’ of file2.
 ‘RdL’ – Delete lines in range ‘R’ from file1 that would have appeared at line ‘L’ in file2

Syntax: diff [options] file1 file2

USAGE OF WinSCP

Introduction to WinSCP

WinSCP (Windows Secure CoPy) is a popular , free and open-source Secure File Transfer
application (SFTP) for Windows computers .It is used to transfer files securely between
remote server and local computer and vice verse. It includes support for SCP file transfers,
and has the ability to create secure tunnels via SSH. Secure FTP (SFTP) is similar to FTP, but
with SFTP the entire session is encrypted, so that passwords are never sent in the clear
text , and are therefore much less vulnerable to interception. That is ,for
transferring files securely, we can use Secure copy or SCP protocol, it uses Secure Shell
(SSH), which encrypts the content before transferring data over network. It runs over TCP
port 22 by default.

How to use WinSCP

Step I : Start WinSCP


To start with double click on WinSCP icon in the desktop or select it from Programs Group .
The
first screen will look as below
Step II : Logging in to Server
Enter IP address of the server in Host Name field.
Unix login userid in the User name field
Unix login Password in Password field
Choose the protocol ( SFTP/ SCP / FTP)
For example
1) Host name used : 172.26.132.40 (For example)
2) User name used : 390119
3) Password used : ********
3) Protocol selected : SCP
Step III : Select/Navigate to working directory
1. We can select the working directory in both Local as well as remote machine from
the drop
down menu.
2. Tool bar can be used to navigate to desired folder
Step IV : Transferring content Content can be transferred in two way
A . Downloading : Transferring content from Remote server to Local machine
B . Uploading : Transferring content from Local machine to Remote server
Following Steps to perform for Downloading file/Directory from Server
Step - A1 : Select the file to transfer in the Remote pane,
Step - A2 : Right click → Select Copy or move option

Step - A3 : Select the destination folder in the local machine(can be changed by


selecting Browse button )
Step - A4 : The file Unix_Recap.txt copied to desktop.

Uploading files to Remote Server


Step - B1 : Select the file to copy in Local machine(Left) pane.
Step - B2 : Drag it and drop it to the remote pane desired location.
Step - B3 : Select the desired location from drop down list and select copy

Saving sessions for future use


All the session information can be saved for future use and faster access.
The information can be saved are :
>> Server IP
>> User name,
>> Password(Not recommended)
>> Communication protocol

Filters

Filter is a programm in unix . It takes its input from another program, performs some operation
on that input, and writes the result to the standard output. Thus the common use of filters is to
modify or restructure output.

Some common filters in UNIX are:

 uniq – Removes identical adjacent lines


 head – displays first n lines of a file .
 tail – displays last n lines of a file .
 sort – sorts files by line (lexically or numerically)
 cut – select portions of a line.
 wc – word count (line count, character count)
 tr – translate
 grep, egrep – search files using regular expressions

head

This command list the beginning of a file to standard output. The default is 10 lines, but a
different number can be specified. The command has a number of options.

Syntax:

head [OPTION] [FILE]

Options:

-c Prints the first N bytes of file; with leading -, prints all but the last N bytes of the file.

-n Prints first N lines; with leading - print all but the last N lines of each file.

Example: To display the first 10 lines of the file myfile.txt. $head myfile.txt

To display the first 100 lines of the file myfile.txt.

$head -n100 myfile.txt

To print the first 5 bytes from the file

$ head -c5 myfile.txt

tail

List the (tail) end of a file to stdout. The default is 10 lines, but this can be changed with the -n
option. Commonly used to keep track of changes to a system log-file, using the -f option, which
outputs lines appended to the file.

Syntax:

tail [OPTION]... [FILE]...

Example:

To display the last 10 lines of the file myfile.txt.


$tail myfile

To display the last 100 lines of the file myfile.txt.

$ tail -100 myfile.txt

$tail –n 100 myfile.txt

more

more command allows to view text files or other output in a scrollable manner. When can
command is used to view a very long file, all the output scrolls off the top of your screen and
only the last page can be viewed. more command solves this problem by allowing the output of
cat command one screenful of data at a time.

Syntax:

more [option] filename

Options:

-num This option specifies an integer which is the screen size (in lines).

-d more will prompt the user with the message "[Press space to continue, 'q' to quit.]" and will
display "[Press 'h' for instructions.]" instead of ringing the bell when an illegal key is pressed.

-l more usually treats ^L (form feed) as a special character, and will pause after any line that
contains a form feed. The -l option will prevent this behavior.

-p Do not scroll. Instead, clear the whole screen and then display the text.

tr
tr command automatically translates or substitute characters.
Syntax:

tr [OPTION] set1 [set2]

Translate, squeeze, and/or delete characters from standard input, writing to standard output.
Options:

c. : complements the set of characters in string.


d. : deletes the characters in set1

s. : replaces repeated characters listed in the set1 with single occurrence


t. : truncates set1

Example: To replace any occurrence of a by x, b by y and c by z in a given string

$echo “about to call “|tr [abc] [xyz]

Output : xyout to zxll

Example: To replace non matching characters

$ echo "Hello"|tr -c e a

Output : aeaaaa

In the above example , except the character “e” other characters are replaced by a

Example: Squeez , we can squeeze more than one occurrence of continuous characters with
single occurrence.

$echo “about to call “|tr – s ‘ ‘

Output : about to call

Above example squeezes two or more blank spaces into one.

sort

sort command reorders the lines of a file in ascending or descending order.

The default order is ascending .

Syntax:

sort -t field_delemeter [OPTION] file1 [file 2]

Options:

-k n sort on the nth field of the line

-t char use char as the field delimiter

-n sort numerically
-r reverse order sort

-u removes repeated lines

-m list merge sorted files in list

Examples:

Below examples will help you to understand sort used with different options:

Example 1:

Consider a file named “list”, which has below data

1, Justin Timberlake, Title 545, Price $7.30

2, Lady Gaga, Title 118, Price $7.30

3, Johnny Cash, Title 482, Price $6.50

4, Elvis Presley, Title 335, Price $7.30

5, John Lennon, Title 271, Price $7.90

To sort on the 2nd field of file named “list” we have to use the below command:

$sort –t’,’ –k 2 list

Note: File list is comma separated file.

Output:

4, Elvis Presley, Title 335, Price $7.30

5, John Lennon, Title 271, Price $7.90

3, Johnny Cash, Title 482, Price $6.50

1, Justin Timberlake, Title 545, Price $7.30

2, Lady Gaga, Title 118, Price $7.30

Example 2: Numerically sorting:


To numerically sort data , option to be used is –n

Suppose list is the name of the file having following data:

19

20

49

00

If we sort it as below:

$sort list

Output is :

19

20

200

49

To get the expected output , the command will be

$sort –n list

Output:

19

20

49
200

Sort can sort multiple files also.

$sort file1 file2 file3 …

Example 3: Numerically sort in reverse order

$sort –nr list

Output :

200

49

20

19

Example 4: Sort the file list removing the repeated lines.

Syntax:

$sort –u filename

File list has following content:

Unix

Unix

Linux

Linux

Solaris

Axis

Axis

$sort –u list
Output:

Unix

Linux

Solaris

Axis

uniq

uniq command is used to suppress the duplicate lines from a file. It discards all the successive
identical lines except one from the input and writes the output.

Syntax:

uniq [option] filename

Options:

-u lists only the lines that are unique

lists only the lines that are duplicates

-c counts the frequency of occurrences

Suppress duplicate lines:

The default behavior of the uniq command is to suppress the duplicate line. Note that, you have
to pass sorted input to the uniq, as it compares only successive lines.

If the lines in the file are not in sorted order, then use the sort command and then pipe the
output to the uniq command.

Count of lines:
The -c option is used to find how many times each line occurs in the file. It prefixes each line with
the count.

Display only duplicate lines:

You can print only the lines that occur more than once in a file using the -d option. The -D option
prints all the duplicate lines.

Skip first N fields in comparison:

the -f option is used to skip the first N columns in comparison. Here the fields are delimited by
the space character.

cut

This command is used for text processing. You can use this command to extract portion of text
from a file by selecting columns.

Syntax:

cut –option filename

Select Column of Characters :

To extract only a desired column from a file use -c option.

The following example displays 2nd character from each line of a file test.txt.

$cut –c2 test.txt

Select Column of Characters using Range :

Range of characters can also be extracted from a file by specifying start and end position
delimited with -.

The following example extracts first 3 characters of each line from a file called test.txt

$cut –c 1-3 test.txt

Select Column of Characters using either Start or End Position :

Either start position or end position can be passed to cut command with -c option.
Following example extracts from 3rd character to end of each line from test.txt file.

$cut –c3- test.txt

To extract 8 characters from the beginning from the file test.txt,

$cut –c-8 test.txt


Select a Specific Field from a File :
Instead of selecting x number of characters you can combine option -f and –d to extract a whole
field.

The option -f specifies which field you want to extract,

The option -d specifies what delimiter that is used in the input file.

The following example displays only first field of each lines from /etc/passwd file using the field
delimiter: (colon). In this case, the 1st field is the username.

$ cut -d':' -f1 etc/passwd

paste

This is the command for merging together different files into a single, multi-column file. In
combination with cut, useful for creating system log files.
Syntax:

paste file1 file2

join
This utility allows merging two files in a meaningful fashion, which essentially creates a simple
version of a relational database.

The command join operates on exactly two files, but pastes together only those lines with a
common tagged field (usually a numerical label), and writes the result to standard output.

The files to be joined should be sorted according to the tagged field for the matchups to work
properly.

Example:
The content of two files file1 and file2 are as below,
$cat file1

100 Shoes

200 Laces

300 Socks

$cat file2

100 $40.0

200 $1.00

300 $2.00

The following command will join these two files.

$ join 1.data 2.data

100 Shoes $40.00

200 Laces $1.00

300 Socks $2.00

Pipe

In unix , you can connect two commands together so that the output from one program
becomes the input of the next program. Two or more commands connected in this way form a
pipe. In shell the symbol '|’ is used to represent pipe.
Purpose of Pipes :

Using pipe you can construct powerful unix command lines by combining basic unix commands.
UNIX commands are powerful; however by using pipe you can combine them together, to
accomplish complex tasks with ease.

Through the standard output of one command (the command to the left of the pipe) gets sent as
standard input to another command (the command to the right of the pipe). Pipe functions in a
similar manner like the output redirection in UNIX (using > symbol to redirect the standard
output of a command to a file. However, the pipe is different because it is used to pass the
output of a command to another command, not a file.

Example:

$ cat apple.txt | wc

3 4 21
In this example, the contents of the file apple.txt are sent through pipe to wc (word count)
command. The wc command then does its job and counts the lines, words, and characters in the
file.

You can combine many commands with pipes on a single command line. Here's an example
where the characters, words, and lines of the file apple.txt is sent to wc and then the output of
wc mailed to nobody@december.com with the subject line "The count."

$ cat apple.txt | wc | mail -s "The count" nobody@december.com

awk

awk is a scripting language which is used for processing or analyzing text files.

awk is used for grouping of data based on either a column or field, or on a set of columns.

It derives its name from the first letter of the last name of its three authors namely Alfred V. Aho,
Peter J.Weinberger and Brian W. Kernighan.
awk can be used for reporting data in a useful manner. It searches one or more files to see if
they contain lines that match specified patterns and then perform associated actions. awk is an
advanced filter.
Simple awk Filtering

Syntax of awk:

~$ awk 'pattern {action}' input-file


Let’s take a input file with the following data
~$cat awk_file
Name,Marks,Max_Marks
Peter,200,1000
Sam,500,1000
Greg,1000
Abharam,800,1000
Henry,600,1000
Peter,400,1000

Example: Default behavior of awk


Print all the lines from a file.

By default, awk prints all lines of a file, so to print every line of above created file , use below
command:
~$ awk '{print}' awk_file

Name,Marks,Max_Marks
Peter,200,1000
Sam,500,1000
Greg,1000
Abharam,800,1000
Henry,600,1000
Peter,400,1000

Example 2: Print only specific field


Print 2nd & 3rd fileds
~$ awk -F”,” {print $2,$3;}' awk_file

Example: Pattern Matching


Print the lines which matches the pattern (lines which contains the word “Henry" or "Peter”)
~$ awk '/Henry|Peter/' awk_file
Peter,200,1000
Henry,600,1000
Peter,400,1000
Initialization and Final Action

BEGIN and END blocks are helpfull in displaying information before and after executing actual
awk script.

BEGIN block is evaluated before awk starts processing the actual awk script; it’s an excellent
place to initialize the FS (field separator) variable, print a heading, or initialize other global
variables.

BEGIN block Usages:

 Declaring variables.
 Initialization variables for doing increment/decrements operations in main AWK code.
 Printing Headings/info before actual AWK code output.

END block is evaluated after all the lines in the awk script have been processed.
Typically, the END block is used to perform final calculations or print summaries that should
appear at the end of the output stream.

END block Usages:

 Printing final results, after doing operations in main AWK block.


 Printing Completion/info after actual AWK code output.

awk tool is mainly used for reporting some data in useful manner. Without these BEGIN and END
blocks the output will be meaningless.
Consider db.txt which contains below data:
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37

awk BEGIN block

This is a block of code which is executed before executing actual awk script.
BEGIN block Syntax
awk ‘BEGIN{awk initializing code}{actual AWK code}’ filename.txt
Example: Print a meaning full info before actual AWK output.
~$ awk ‘BEGIN{print “########################\nThis is the output of
filtered
data\n########################”}{print $0}’ db.txt

Output:
##########################
This is the output of filtered data
##########################
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
Edwin 253734 87 97 95
Dayan 24155 30 47

awk END block

This is the block which is executed after executing all the awk code.
Example:

Print some meaning full info after processing awk code.


~$ awk ‘{print $0} END {print “#########################\n Completed
printing
filtered data\n########################”}’ db.txt

Output:

Jones 21 78 84 77
Gondrol 23 56 58 45
RinRao 25 21 38 37
Edwin 25 87 97 95
Dayan 24 55 30 47
#########################
Completed printing filtered data
#########################

Combining BEGIN and END block

Example:

~$ awk ‘BEGIN{print “##########################\n This is the output of filtered


data\n##########################”}{print $0}END{print
“########################\n Completed printing filtered
data\n########################”}’ db.txt

Output:

#########################
This is the output of filtered data
#########################
Jones 21 78 84 77
Gondrol 23 56 58 45
RinRao 25 21 38 37
Edwin 25 87 97 95
Dayan 24 55 30 47
########################
Completed printing filtered data

awk inbuilt variables

awk is supplied with good number of built-in variables which comes in handy when working with
data files. We will see usages of awk built-in variables with one or two examples . These variable
are used to format the output of an awk command.

List of built-in variables:

FS field separator character (default blank & tab)


OFS output field separator string (default blank)
RS input record separator character (default newline)
ORS output record separator string (default newline)
NF number of fields in input record
NR number of input record
FNR output number of lines
FILENAME name of current input file

Consider below db.txt as sample file.


~$ cat db.txt
John,29,MS,IBM,M,Married
Barbi,45,MD,JHH,F,Single
Mitch,33,BS,BofA,M,Single
Tim,39,Phd,DELL,M,Married
Lisa,22,BS,SmartDrive,F,Married

In order to make it simple we can divide above inbuilt variables in to groups on basis of their
operations.
Group1: FS(input field separator), OFS(Output Field Separator)
Group2: RS(Row separator) and ORS(Output Record Separator)
Group3: NR, NF and FNR
Group4: FILENAME variable

FS (Input Field Separator)


This variable is useful in storing the input field separator. By default AWK can understand only
spaces, tabs as input and output separators. But if your file contains some other character as
separator other than these mention one’s, awk cannot understand them.

For example UNIX password file which contain ‘:’ as a separator. So in order to mention the input
filed separator we use this inbuilt variable. We will see what issue we face if we don’t mention
the field separator for our db.txt.

Example: without using FS

Print first column data from db.txt file.


~$ awk ‘{print $1}’ db.txt

Output:
John,29,MS,IBM,M,Married
Barbi,45,MD,JHH,F,Single
Mitch,33,BS,BofA,M,Single
Tim,39,Phd,DELL,M,Married
Lisa,22,BS,SmartDrive,F,Married

OFS (Output Field Separator)


This variable is useful for defining the output field separator for the expected output data.

Example:

Display only 1st and 4th column and with $ as field separator for the output .
~$ awk ‘BEGIN{FS=”,”;OFS=” $ “}{print $1,$4}’ db.txt

Output:
John $ IBM
Barbi $ JHH
Mitch $ BofA
Tim $ DELL
Lisa $ SmartDrive
Note: Space is give before and after $ in OFS variable to show better output.

RS (Row separator)
Row Separator is helpful in defining separator between rows in a file. By default awk takes row
separator as new line. We can change this by using RS built-in variable.
Example:
Convert a sentence to a word per line. We can use RS variable for doing it.
~$ echo “This is how it works” | awk ‘BEGIN{RS=” ”}{print $0}’

Output:
This
is
how
it
Works

ORS (Output Record Separator)


This variable is useful for defining the record separator for the awk command output. By default
ORS is set to new line.

Example:
Print all the company names in single line which are in 4th column.
~$ awk -F’,’ ‘BEGIN{ORS=” “}{print $4}’ db.txt

Output:
IBM JHH BofA DELL SmartDrive

NF
This variable keeps information about total fields in a given row. The final
value of a row can be represented with $NF.

Example: Consider abc.txt which contains below data:


Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
Edwin 253734 87 97 95
Dayan 24155 30 47
Print number of fields in each row in abc.txt.
~$ awk ‘{print NF}’ abc.txt

Output:
5
5
4
5
4
NR

This variable keeps the value of present line number. This will come handy when you want to
print line numbers in a file.

Example:
Print line number for each line in a given file.
~$ awk ‘{print NR, $0}’ abc.txt

Output:
1 Jones 2143 78 84 77
2 Gondrol 2321 56 58 45
3 RinRao 2122234 38 37
4 Edwin 253734 87 97 95
5 Dayan 24155 30 47
This can be treated as cat command -n option for displaying line number for a file

FNR
This variable keeps count of number of lines present in a given file/data. This will come handy
when
you want to print no of line present in a given file. This command is equivalent to wc -l
command.

Example:
Print total number of lines in a given file.
~$ awk ‘END{print FNR}’ abc.txt

Output:
5
FILENAME
This variable contain file awk command is processing.

Example:
Print filename for each line in a given file.
~$ awk ‘{print FILENAME, NR, $0}’ abc.txt

Output:
abc.txt 1 Jones 2143 78 84 77
abc.txt 2 Gondrol 2321 56 58 45
abc.txt 3 RinRao 2122234 38 37
abc.txt 4 Edwin 253734 87 97 95
abc.txt 5 Dayan 24155 30 47

awk Built in Function

A function is a self-contained computation that accepts a number of arguments as input and


returns some value. awk has a number of built-in functions in two groups: arithmetic and string
functions.

Arithmetic Functions
Nine of the built-in functions can be classified as arithmetic functions. Most of them take a
numeric argument and return a numeric value. Below table summarizes these arithmetic
functions with some Examples.

awk Function Description


cos ( x ) Returns cosine of x (x is in radians).
exp ( x ) Returns e to the power x.
index (s1,s2) Position of string s2 in s1; returns 0 if not present
int ( x ) Returns truncated value of x.
log ( x ) Returns natural logarithm (base- e) of x.
sin ( x ) Returns sine of x (x is in radians)

sqrt ( x ) Returns square root of x.


atan2 ( y , x ) Returns arctangent of y / x in the range - to .
rand () Returns pseudo-random number r, where 0 <= r < 1.
sqrt(expr) Returns the square root of the expression or value given

Examples:
~$ awk 'BEGIN{
print sqrt(16);
print sqrt(0);
print sqrt(-12);
}'
Output:
4
0
nan
Here nan stands for not a valid number.
String Functions
The built-in string functions are much more significant and interesting than the numeric
functions. Because awk is essentially designed as a string-processing language, a lot of its power
derives from these functions. Below table lists the string functions found in awk. awk's Built-In
String Functions

Filters and Regular Expression

grep
grep command allows you to search one file or multiple files for lines that contain a pattern.

Full form of grep is global regular expression print.


It is a powerful file pattern searcher in Linux

grep's exit status is 0 if matches were found, 1 if no matches were found, and 2 if errors
occurred.
grep search the target file(s) for occurrences of pattern, where pattern may be literal text or a
Regular Expression.

Syntax:
grep pattern [file...]

Search for the given string in a single file


The basic usage of grep command is to search for a specific string in the specified file as shown
below.

Checking given string in multiple files


We can use grep command for searching for a given string in multiple files.

For example, let us copy the demo_file to demo_file1and use the grep on both the files to search
the pattern this.

The output will include the file name in front of the line that matched the specific pattern as
shown below.

When the Linux shell sees the meta character, it does the expansion and gives all the files as
input to grep.
Case insensitive search
We can use grep to search for the given string/pattern case insensitively. So it matches all the
words such as “the”, “THE” and “The” case insensitively as shown below.

Match regular expression in files


This is a very powerful feature of grep . In the following example, it searches for all the pattern
that starts with “lines” and ends with “empty” with anything in-between. i.e To search
“lines[anything in-between]empty” in the demo_file.

A regular expression may be followed by one of several repetition operators:

1. ? The preceding item is optional and matched at most once.


2. The preceding item will be matched zero or more times.
3. + The preceding item will be matched one or more times.
4. {n} The preceding item is matched exactly n times.
5. {n,} The preceding item is matched n or more times.
6. {,m} The preceding item is matched at most m times.
7. {n,m} The preceding item is matched at least n times, but not more than m times.

Checking for full words


To search for a word, and to avoid it to match the substrings -w option is used. The following
example is the regular grep where it is searching for “is”. When you search for “is”, without any
option it will show out “is”, “his”, “this” and everything which has the substring “is”.
Searching in all files recursively
When you want to search in all the files under the current directory and its sub directory ‘–r’
option is the one which you need to use. The following example will look for the string “ramesh”
in all the files in the current directory and all its subdirectory.

$ grep -r "ramesh" *

Invert match
If you want to display the lines which does not matches the given string/pattern, use the option -
v as shown below. This example will display all the lines that did not match the word “Two”.

Displaying the lines which does not matches the entire given pattern.

Syntax:
grep -v -e pattern -e pattern

For example, the file file1 has the following content

Apple

Banana
Cauliflower
Grapes
Orange
Counting the number of matches
Count the number of lines matched in the given pattern/string, then use the option -c.
Syntax:
grep -c pattern filename

Displaying only the file names which matches the given pattern
The -l option is used to display only the file names which matched the given pattern. When you
give multiple files to the grep as input, it displays the names of file which contains the text that
matches the pattern, will be very handy when you try to find some notes in your whole directory
structure.

Showing line number while displaying the output


To show the line number of file with the line matched, -n option is used.

Syntax:
grep -n pattern filename

Example:
grep -n "this" demo_file
2: this line is the 1st lower case line in this file.
6: Two lines above this line is empty.

sed

sed is a stream editor used to perform basic text transformations on an input stream (a file, or
input from a pipeline).

Working methodology
sed works by making only one pass over the input(s) s called as one execution cycle. Cycle
continues till end of file/input is reached.

 Read entire line from stdin/file.


 Removes any trailing newline.
 Places the line, in its pattern buffer.
 Modify the pattern buffer according to the supplied commands.
 Print the pattern buffer to stdout.

Printing Operation in sed


sed allows you to print only specific lines based on the line number or pattern matches. “p” is
the command for printing the data from the pattern buffer. To suppress automatic printing of
patternspace -n option is used with sed. sed -n option will not print anything, unless an explicit
request to print is found.

Syntax:
sed -n 'ADDRESS'p filename
sed -n '/pattern/p' filename

Examples:
Let us assume the demo_file has the following content

To prints third line of input file

$sed -n '3p' demo_file


3. Hardware

To print every nth line starting from the line m


$sed -n 'm~np' filename

To print only the last line

To print the lines containing the given pattern:


Syntax:
sed -n /PATTERN/p filename
Deletion operation in sed
In sed the d command is used to delete the pattern space buffer and immediately starts the next
cycle.
Syntax:
sed nd filename
'nd’ deletes the nth line and prints the other lines.
sed 'ADDRESS'd filename
sed /PATTERN/d filename

The process is
• It reads the first line and places in its pattern buffer
• checks whether supplied command is true for this line , if true, deletes pattern space buffer
and starts next cycle and reads the next line.
• If supplied command is not true, it prints the content of the pattern space buffer.

To delete the 3rd line and print other lines from the file demo_file

Substitution operation in sed


In sed the s command is used to substitute the pattern. The `s’ command attempts to match the
pat-tern space against the supplied expression/ pattern; if the match is successful, then that
portion of the pattern space which was matched is replaced with the replacement given.

Syntax:
$sed 'ADDRESSs/REGEXP/REPLACEMENT/FLAGS' filename
$sed 'PATTERNs/REGEXP/REPLACEMENT/FLAGS' filename

1. s is substitute command
2. / is a delimiter
3. REGEXP is regular expression to match
4. REPLACEMENT is a value to replace

FLAGS can be any of the following:


1. g Replace all the instance of REGEXP with REPLACEMENT
2. n Could be any number,replace nth instance of the REGEXP with REPLACEMENT.
3. p If substitution was made, then prints the new pattern space.
4. i match REGEXP in a case-insensitive manner.
5. w file If substitution was made, write out the result to the given file.
6. We can use different delimiters ( one of @ % ; : ) instead of /

To Write Changes to a File and Print the Changes

To combine multiple sed commands we have to use option -e

Syntax:
$sed -e 'command' e 'command' filename

To Delete the first,last and all the blank lines from input

FILTERS USING REGULAR EXPRESSION


A regular expression is a set of characters that specify a pattern. Regular expressions are used
when you want to search for specific lines of text containing a particular pattern. Most of the
UNIX utilities operate on ASCII files a line at a time. Regular expressions search for patterns on a
single line, and not for patterns that start on one line and end on another.

The Structure of a Regular Expression

There are three important parts to a regular expression.


• Anchors : These are used to specify the position of the pattern in relation to a line of text.
• Character Sets : The set of characters that match one or more characters in a single position.
• Modifiers: They specify how many times the previous character set is repeated.
A simple example that demonstrates all three parts is the regular expression is : "^#*"
Here ,
• The up arrow , “^”, is an anchor that indicates the beginning of the line.
• The character "#" is a simple character set that matches the single
character "#".
• The asterisk “*” is a modifier. In a regular expression it specifies that the
previous character set can appear any number of times, including zero.

There are also two types of regular expressions:


• the "Basic" regular expression,(BRE)
• the "extended" regular expression.(ERE)

A few utilities like awk and egrep use the extended expression. Most use the "regular" regular
expression. From now on, if I talk about a "regular expression," it describes a feature in both
types.

The Anchor Characters: ^ and $ Anchores are used when we want to search for a pattern that is
at one end or the other, of a line. The character "^" is the starting anchor, and the character "$"
is the end anchor.Following list provides a summary:

Pattern Matches
^A "A" at the beginning of a line
A$ "A" at the end of a line
A^ "A^" anywhere on a line
$A "$A" anywhere on a line
^^ "^" at the beginning of a line
$$ "$" at the end of a line
The Character Set
The character set also called “character class” in a regular expression , is used to tell the regex
engine to match only one out of several characters

 A character set matches only a single character. In case of the above example, gr[ae]y
does not match graay, graey or any such thing.
 The order of the characters inside a character set does not matter. The results are
identical.
 Some characters have a special meaning in regular expressions. If we want to search for
such a character, we have to escape it with a backslash.

Exception in the character class


If we want to search for all the characters except those in the square bracket,
then the ^ (Caret) symbol needs to be used as the first character after open
square bracket. The expression "^[^aeiou]" is to searc for a line which does
not start with the vowel letter.

Regular Expression Matches


[] The characters "[]"
[0] The character "0"
[0-9] Any number
[^0-9] Any character other than a number
[-0-9] Any number or a "-"
[0-9-] Any number or a "-"
[^-0-9] Any character except a number or a "-"
[]0-9] Any number or a "]"
[0-9]] Any number followed by a "]"
[0-9-z] Any number, or any character between "9" and "z".
[0-9\-a\]] Any number, or a "-", a "a", or a "]"

Match any character


The character "." is one one of thespecial meta-characters. By itself it will match any character,
except the end-of-line character. Thus the pattern that will match a line with a single characters
is ^.$

Repeating character sets


The third part of a regular expression is the modifier. It is used to specify how may times you
expect to see the previous character set. The repetition modifier * find no or one, one or more,
and zero or more
repeats, respectively.

Examples:
Expression Matches
Go*gle Gogle,Google,Gooogle, and so on.
"[0-9]*" zero or more numbers.

Matching a specific number of sets with \{ and \}


We cannot specify a maximum number of sets with the "*" modifier. There is a special pattern
we can use to specify the minimum and maximum number of repeats, by putting those two
numbers between "\{" and "\}".

▪ A modifier can specify amounts such as none, one, or more;

For example , A user name is a string beginning with a letter followed by at least two, but not
more than seven letters or numbers followed by the end of the string. Then the regular
expression is

^[A-z][A-z0-9]{2,7}

▪ A repetition modifier must be combined with other patterns; the modifier has no meaning by
itself.

For example , modifiers like "*" and "\{1,5\}" only act as modifiers if they follow a character set. If
they were at the beginning of a pattern, they would not be a modifier.

grep with Regular expression

Search for 'vivek' in /etc/passswd


grep vivek /etc/passwd
Search vivek in any case (i.e. case insensitive search)
grep -i -w vivek /etc/passwd
Search vivek or raj in any case
grep -E -i -w 'vivek|raj' /etc/passwd

Line and word anchors

Search lines starting with the vivek only


grep ^vivek /etc/passwd
To display only lines starting with the word vivek only i.e. do not display vivekgite, vivekg
grep -w ^vivek /etc/passwd
To Find lines ending with word foo
grep 'foo$' filename
Character classes

To match Vivek or vivek.


grep '[vV]ivek' filename
OR
grep '[vV][iI][Vv][Ee][kK]' filename

To match digits (i.e match vivek1 or Vivek2 etc)


grep -w '[vV]ivek[0-9]' filename

Wildcards
To match all 3 character word starting with "b" and ending in "t".
grep '\' filename
Where,
•\< Match the empty string at the beginning of word
•\> Match the empty string at the end of word.
Print all lines with exactly two characters
grep '^..$' filename
Display any lines starting with a dot and digit
grep '^\.[0-9]' filename

Escaping the dot

To find an IP address 192.168.1.254


grep '192\.168\.1\.254' /etc/hosts

Search a Pattern Which Has a Leading – Symbol

Searches for all lines matching '--test--' using -e option . Without -e, grep would
attempt to parse '--test--' as a list of options
grep -e '--test--' filename

Test Sequence

To Match a character "v" two times


egrep "v{2}" filename
To match both "col" and "cool"
egrep 'co{1,2}l' filename

grep OR Operator
Suppose the file “employee” has the following data:

To find the records of those who are either from Tech or Sales dept.
We can use the following syntaxes :

1) Syntax : grep 'word1\|word2' filename


grep 'Tech\|Sales' employee

2) Syntax : grep -E 'pattern1|pattern2' fileName


grep -E 'Tech|Sales' employee

grep AND Operator


There is no AND operator in grep. But, we can simulate AND using

• grep -E option.
Syntax : grep -E 'word1.*word2 ' filename
grep -E 'word1.*word2|'word2.*word1' filename

• multiple grep command separated by pipe


Syntax : grep 'word1' filename | grep 'word2'

Shell Programming

The UNIX shell program interprets user commands, which are either directly entered by the user,
or which can be read from a file called the shell script or shell program. Shell scripts are
interpreted, not compiled. The shell reads commands from the script line by line and searches
for those commands on the system.

Shell script
Shell script is a file-containing list of commands to be executed in a particular order.

A good shell script will have comments, (a statement preceded by a pound sign, #) , describing
the purpose of the statement.

In a script we can use conditional tests, such as value A is greater than value B, loops or iterative
statements to execute some steps repetitively or to navigate through a list of data or records
and do processing. We can use files to read and store data. Can use variables to read and store
data. A script may include functions also.

When a script is executed, the shell reads the commands one by one and executes them .

We can create the simple shell script file by using vi editor or cat command like,

$ vi test.sh

$ cat > test.sh

Below mentioned shebang statement should be the first statement in the shell script as it tells
the system that the commands mentioned in the shell script are to be executed by the shell
/bin/sh

#!/bin/sh

Consider the shell script with just two commands pwd & ls.

$cat test.sh

#!/bin/bash

pwd

ls

Importance of shell script

Shell scripts are basically used for automating processes that we repeat at the prompt .

Following are some activities we can do using shell scripts:

 Automation of repetitive task


 Creating our own power tools/utilities.
 Automating command input or entry.
 Customizing administrative tasks.
 Creating simple applications.
 Automating administration tasks such as adding new users, removing obsolete users etc

Some Practical examples where shell scripting can be actively used:

 Monitoring your Linux system.


 Data backup and creating snapshots.
 Dumping Oracle or MySQL database for backup.
 Creating email based alert system.
 Find out what processes are eating up your system resources.
 Find out available and free memory.
 Find out all logged in users and what they are doing.
 Find out if all necessary network services are running or not. For example if web server
failed then send an alert to system administrator via a pager or an email.
 Find out all failed login attemp. If login attempt are continued repeatedly from same
network IP, automatically block all those IPs accessing your network/service via firewall.
 User administration as per your own security policies.
 Find out information about local or remote servers.
 Configure server such as BIND (DNS server) to add zone entries.

Executing a shell script

Below two methods can be used to execute the shell script.

$ sh filename

Or

$ ./filename

In this case we have to modify the file access permissions of the shell script before execution.

To provide execute permission , following command is used.

$ chmod u+x filename

Basic Operators in shell scripting

Below operators are supported by shell.

 Arithmetic Operators.
 Relational Operators.

 Boolean Operators.

 String Operators.

 File Test Operators.

Here is simple example to add two numbers:

Example:

#!/bin/sh

val=`expr 2 + 2`

echo "Total value : $val"

Output:

$ Total value : 4

There are following points to note down:

 There must be spaces between operators and expressions for example 2+2 is not correct,
where as it should be written as 2 + 2.

 Complete expression should be enclosed between ``, called inverted commas to execute
expr command correctly.

Arithmetic Operators

Assume variable a holds 10 and variable b holds 20 then:

Operator Description Example


+ Addition `expr $a + $b ` will give 30
- Substraction `expr $a - $b ` will give -10
* Multiplication `expr $a \* $b` will give 200
/ Division `expr $b / $a` will give 2
% Modulus `expr $a % $b` will give 0
!= Not equal [ $a != $b ] will give true
= assignment a=$b will assign value of b to a.
== Equality [ $a == $b ] will return false.

It is very important to note here that all the conditional expressions would be put inside square
braces with one spaces around them, for example [ $a == $b ] is correct where as [$a==$b] is
incorrect.

Relational Operators

Below are relational operators which are specific to numeric values. These operators would not
work for string values unless their value is numeric.

For example, following operators would work to check a relation between 10 and 20 as well as in
between "10" and "20" but not in between "ten" and "twenty".

Assume variable a holds 10 and variable b holds 20 then:

Operator Description Example

-eq Check if the values of 2 operands are equal or not, if yes [ $a -eq $b ] is false
then condition becomes true.

-ne Check if the values of 2 operands are equal or not, if [ $a -eq $b ] is true
values are not equal then condition becomes true.
-gt Check if the value of left operand is greater than the value [ $a -gt $b ] is false
of right operand, if yes then condition becomes true.
-lt Check if the value of left operand is less than the value of [ $a -lt $b ] is true
right operand, if yes then condition becomes true.
-ge Check if the value of left operand is greater than or equal [ $a -ge $b ] is false
to the value of right operand, if yes then condition
becomes true.
-le Check if the value of left operand is less than or equal to [ $a -le $b ] is true
the value of right operand, if yes then condition becomes
true.

It is very important to note here that all the conditional expressions would be put inside square
braces with one spaces around them, for example [ $a <= $b ] is correct where as [$a <= $b] is
incorrect.

Boolean Operators

Assume variable a holds 10 and variable b holds 20 then

Operator Description Example


! This is logical negation. This inverts a true [ !false ] is true
condition into false and vice versa.
-o This is logical OR. If one of the operands is true [ $a -lt 20 -o $b -gt 100 ] is true
then condition would be true.
-a This is logical AND. If both the operands are true [ $a -lt 20 -a $b -gt 100 ] is
then condition would be true otherwise it would false.
be false.

String Operators

These are string operators. Assume variable a holds "abc" and variable b holds "efg" then:

Operator Description Example


-z Check if the given string operand size is zero. If it is [ -z $a ] will return false.
zero length then it returns true.
-n Check if the given string operand size is non- zero. [ -z $a ] will return true.
If it is non-zero length then it returns true.
= Check if the value of two operands is equal or not, [ $a = $b ] will return false
if yes then condition becomes true.
!= Check if the value of two operands is equal or not, [ $a != $b ] will return true
if the values are not equal then condition becomes
true.
str Check if the str is not the empty string. If it is empty [ $a ] will return true
then it returns false.
File Test Operators:

Assume a variable file holds an existing file name "test" whose size is 100 bytes and has read,
write and execute permission on:

Operator Description Example


-b file Returns true, if file is a block special file [ -b $file ] is false.
-c file Returns true, if file is a character special file [ -b $file ] is false.
-d file Returns true, Check if file is a directory [ -d $file ] is not true.
-f file Returns true, Check if file is an ordinary file or [ -f $file ] is true.
special file
-r file Returns true, Checks if file is readable [ -r $file ] is true.
-w file Returns true, Check if file is writable [ -w $file ] is true.
-x file Returns true, Check if file is execute [ -x $file ] is true.
-s file Returns true, Check if file has size greater than 0 [ -s $file ] is true.
-e file Returns true, Check if file exists [ -e $file ] is true.

Wild Card Characters

Symbol used to replace or represent one or more characters. Wildcards or wild characters are
either asterisk (*), which represent one or more characters or question mark (?), which
represent a single character.

Wild card Meaning Examples


/Shorthand
Matches any string or $ ls * will show all files
group of characters. $ ls a* will show all files whose first name is starting
with letter 'a'
*
$ ls *.c will show all files having extension .c
$ ls ut*.c Will show all files having extension .c but file
name must begin with 'ut'.
? Matches any single $ ls ? will show all files whose names are 1
character. character long
$ ls fo? will show all files whose names are 3
character long and file name begin with fo
[...] Matches any one of the $ ls [abc]* Will show all files beginning with letters a,b,c
enclosed characters

Note: [..-..] A pair of characters separated by a minus sign denotes a range.

Example:

$ ls /bin/[a-c]*

Will show all File name beginning with letter a,b or c.

Output:

/bin/arch /bin/awk /bin/bsh /bin/chmod /bin/cp /bin/as /bin/basename /bin/cat

Shell Quoting Mechanism

The Metacharacters

Unix Shell provides various metacharacters which have special meaning while using them in any
Shell Script and causes termination of a word unless quoted.

Example:

? Matches with a single character while listing files in a directory and an * would match more
than one characters.

Here is a list of most of the shell special characters (also called metacharacters):

* ? [ ] ' " \ $ ; & ( ) | ^ < > new-line space tab

A character may be quoted (i.e., made to stand for itself) by preceding it with a \.

Example:

#!/bin/sh echo Hello; Word

This would produce following result.

Hello ./test.sh: line 2: Word: command not found shell returned 127

Now let us try using a quoted character:

#!/bin/sh echo Hello\; Word


This would produce following result:

Hello; Word

The $ sign is one of the metacharacters, so it must be quoted to avoid special handling by the
shell:

#!/bin/sh echo "I have \$1200"

This would produce following result:

I have $1200

Quote Description
Single quote All special characters between these quotes lose their special meaning.

Double quote Most special characters between these quotes lose their special meaning with
these exceptions:

\$

\'

\"

\\
Backslash Any character immediately following the backslash loses its special meaning.

Back Quote Anything in between back quotes would be treated as a command and would be
executed.

The Single Quotes

Consider an echo command that contains many special shell characters:

echo <-$1500.**>; (update?) [y|n]

Putting a backslash in front of each special character is tedious and makes the line difficult to
read:
echo \<-\$1500.\*\*\>\; update\?update\?

y∥ny‖n

There is an easy way to quote a large group of characters. Put a single quote ( ') at the beginning
and at the end of the string:

echo '<-$1500.**>; (update?) [y|n]'

Any characters within single quotes are quoted just as if a backslash is in front of each character.
So now this echo command displays properly.

If a single quote appears within a string to be output, you should not put the whole string within
single quotes instead you would precede that using a backslash (\) as follows:

echo 'It\'s Shell Programming

The Double Quotes:

Try to execute the following shell script. This shell script makes use of single quote:

VAR=ZARA

echo '$VAR owes <-$1500.**>; [ as of (`date +%m/%d`) ]'

This would produce following result:

$VAR owes <-$1500.**>; [ as of (`date +%m/%d`) ]

So this is not what you wanted to display. It is obvious that single quotes prevent variable
substitution. If you want to substitute variable values and to make invert commas work as
expected then you would need to put your commands in double quotes as follows:

VAR=ZARA

echo "$VAR owes <-\$1500.**>; [ as of (`date +%m/%d`) ]"

Now this would produce following result:

ZARA owes <-$1500.**>; [ as of (07/02) ]

Double quotes take away the special meaning of all characters except the following:

 $ for parameter substitution.


 Backquotes for command substitution.
 \$ to enable literal dollar signs.
 \` to enable literal backquotes.
 \" to enable embedded double quotes.
 \\ to enable embedded backslashes.
 All other \ characters are literal (not special).

Back Quotes: Command Substitution

Putting any Shell command in between back quotes would execute the command

Syntax: var=`command`

Example:

Following would execute date command and produced result would be stored in DATA variable.

DATE=`date` echo "Current Date: $DATE"

This would produce following result:

Current Date: Thu Jul

Shell Decision Statement

While writing a shell script, there may be situations when you need to adopt one path out of
many available paths. In such cases you need to make use of conditional statements that allow
your program to make correct decisions and perform right actions.

Unix Shell supports conditional statements, which are used to perform different actions based
on different conditions.

Here we will explain following two decision-making statements

 The if...else statement


 The case...esac statement

If..else statements

We can use “if..else” statement which can be used as decision making statement to select an
option from a given set of options.

Unix Shell supports following forms of if..else statement:


 if...fi statement
 if...else...fi statement
 if...elif...else...fi statement

Syntax:

 if...fi statement

if [condition]

then

command(s)

fi

 if...else...fi statement

if [ condition(s) ] then

command(s)

else

command(s)

fi

 if...elif...else...fi statement

if [ condition(s) ]

then

command(s)

elif [ condition(s) ] # Many elif-then allowed

then

command(s)
else

command(s)

fi

test command in if condition

We can use test command as condition of if condition as used in the below script.

$cat if_test.sh

#!/bin/sh

echo “Do you want to quit (Y/y)? ”

read ans

if test $ans ==’y’ –o $ans==’Y’

then

exit

else

echo “ to exit enter N or n”

fi

case...esac Statement

We can use multiple if...elif statements to perform a multiway branch. However, this is not
always the best solution, especially when all of the branches depend on the value of a single
variable.

Unix Shell supports case...esac statement which handles exactly this situation, and it does so
more efficiently than repeated if...elif statements.

The interpreter checks each case against the value of the expression until it finds a match. If
nothing matches, goes with the default condition.

Syntax:
case word in pattern1)

Statement(s) to be executed if pattern1 matches

;;

pattern2)

Statement(s) to be executed if pattern2 matches

;;

pattern3)

Statement(s) to be executed if pattern3 matches

;;

esac

Here string word is compared against every pattern until a match is found and the statement(s)
following the match pattern executes. If shell cannot find any match, the case statement exits
without performing any action. When statement(s) part is executed. The command ;; indicates
program flow should jump to the end of the entire case statement.

There is no maximum number of patterns, but the minimum is one.

Example:

#!/bin/sh

COURSE=”DB”

case “$COURSE” in

“Java”) echo “Java is a programming language”

;;

“Perl”)echo “Perl is scripting language”

;;
“DB”)echo “Oracle is a DB”

;;

esac

Output:

Oracle is a DB

Iterative Statements/Loop :

Loops are a powerful programming tool that enables you to execute a set of commands
repeatedly.

 while loop
 for loop
 until loop

while loop

Here condition is evaluated by shell at the beginning of each iteration. If the resulting value is
true, given command(s) are executed. If condition is false then no command would be executed
and program would jump to the next line after done statement.

Syntax:

while condition

do # executed as long as the condition is true

command(s)

done

Example:

a=0

while [ $a -lt 3 ]

do
echo $a

a=`expr $a + 1`

done

Output:

until loop

Here condition is evaluated by shell. If the resulting value is false, given command(s) are
executed. If condition is true then no command would be executed and program would jump to
the next line after done statement.

Syntax:

until condition # complement of while

do

# executes the body as long as the condition is false

command(s)

done

Example:

a=0

until [ ! $a -lt 3 ] do

echo $a a=`expr $a + 1`

done
Output:

for loop

For loop operate on lists of items. It repeats a set of commands for every item in a list.

Syntax:

for var in w1 w2 w3.. wn

do

command #Statement(s) to be executed for every w in the list

done

Example:

for var in 0 1 2 3 4 5

do

echo $var

done

Output:

5
Example:

for filename in `ls tempdir`

do

echo “Displaying contents of ${filename}”

cat ${filename}

done

We can use "break" and "continue" commands to control the loop.

Command "break" causes the termination of a loop and “continue” resumes execution at its top.

String Handling

 String handling with test command:

test str Returns true if str is not null

test –n str Returns true if length of str is greater than


zero

test –z str Returns true if length of str is equal to zero

 String handling with expr command

The expr is quite handy for finding the length of a string and extracting a sub-string:

Length of the string:

$ str=”abcdefghijk” ;

$ n=`expr "$str" : ‘.*’` ;


$ echo $n

11

expr gave how many times any character (.*) occurs. This feature is very useful in validating data
entry.

Extracting a sub-string:

$ str=”abcdefghijk” ;

$ expr “$str” : ‘……....’

gh

Note that there are 6 dots preceding the sequence ..... This advanced regular expression signifies
that the first six characters of the string are to be ignored and extraction should start from the
7th character. Two dots inside .... suggests that this extraction is limited to two characters only
(backslashes override the usual interpretation of ‘()’).

Extracting string from 3rd character to end of the string:

$ str="abcdefghijk"

$ expr "$str" : '...∗.∗'

cdefghijk

Location of first occurrence of a character “d” inside string:

$ str=”abcdefghijk” ;

$ expr "$str" : '[^d]*d‘

Location of last occurrence if a character inside string:


Below will give the last occurrence of character 'a' from string str.

$str=”abc def abc”

$expr "$str" : '[^u]*a'

9.

6. Command Line Arguments

To make a shell script a generalized script, we should avoid hard coding. User should be able to
provide the values required for processing as input while running the shell script. To facilitate
this we have to use command line arguments.

The statement we write in the command prompt starting with the command followed by list of
arguments is the command line argument.

Example:

$ ls dir1 dir2 dir3

A set of shell variables are there to represent the command line arguments.

$0 – represents the command name (first word on command line). For above example “ls”

$1 - the first argument of command (second word/symbol ).For the above example dir1

$2 – the second argument to the command. For the above example, dir2

In this way we can use up to $9

The command-line arguments $1, $2, $3...$9 are also called positional parameters, with $0
pointing to the actual command/program/shell script and $1, $2, $3, ...$9 as the arguments to
the command.

Consider below sample shell script “test.sh”

$cat test.sh
#!/bin/sh

echo “Program name is $0”


echo “First argument is $1”

echo “Number of arguments passed =$#”

echo “The arguments are $@”

Let’s execute the above shell script “test.sh”. Remember to change File access permission of
script to execute

$ sh test.sh arg1 arg2 arg3

Program name is test

First argument is arg1

Second argument is arg2

Number of arguments passed = 3

The arguments are arg1 arg2 arg3

There’s another way to execute the script using ./

$ ./test.sh arg1 arg2 arg3

Program name is test

First argument is arg1

Second argument is arg2

Number of arguments passed = 3

The arguments are arg1 arg2 arg3

User Input: read

To get input from the keyboard, you use the read command. The read command takes input
from the keyboard and assigns it to a variable.
Example: read.sh

#!/bin/sh

echo –n “Enter some text > “

read text

echo “You entered: $text”

Note that “-n” given to the echo command causes it to keep the cursor on the same line; i.e., it
does not output a carriage return at the end of the prompt.

Next, we invoke the read command with “text” as its argument. What this does is wait for the
user ro type something followed by a carriage return (the Enter key) and then assign whatever
was typed to the variable text.

Execution:

$sh read.sh

Enter some text > This is some text

You entered: This is some text

Validation of Command Line Arguments

Let us assume, we have one shell script which requires exactly 2 arguments to execute. Shell
script should throw proper error message in case user has not given exactly two arguments.

Let us have a small shell script "test.sh" to show how to implement validations.

#!/bin/sh

if [ $# -ne 2 ]

then

echo " Not sufficient arguments, please give exact 2 arguments"


else

echo "Continue with our task" fi

$ ./test.sh

Not sufficient arguments, please give exact 2 arguments

$ sh test.sh abc

Not sufficient arguments, please give exact 2 arguments

$ ./test.sh abc def ghi

Not sufficient arguments, please give exact 2 arguments

$ ./test.sh abc def 2

Not sufficient arguments, please give exact 2 arguments

$ ./test.sh abc def

Continue with our task

For scripts where input expected is a file , validations can be incorporated for file tests like
whether the file given as input exists or not, whether it is empty or not, whether it is readable or
not and so on as per requirements.

Environment Variables:

An important concept in Unix is the environment, which is defined by values stored in a set of
predefined variables called environment variables.

Values of some of these variables are set by the system, some can be set by the user, some of
them by the shell, or any program that loads another program.
When a user logs in to the system, the shell undergoes a twostep process called initialization to
set up the environment. This process involves the shell reading two files −/etc/profile and .profile
in the home directory of the user.

The file /etc/profile is maintained by the system administrator of the Unix machine and contains
shell initialization information required by all users on a system.

The file .profile is under the control of the user. User can add as much shell customization
information as he/she wants to this file. The minimum set of information that a user need to
configure includes −

 The type of terminal user is using.


 A list of directories in which to locate the commands.
 A list of variables affecting the look and feel of the terminal.

Environment around the UNIX Process:

When a process is created, UNIX opens 3 streams stdin/stdout/stderr for basic communication
with respect to the process control terminal. In addition, it knows the current working directory
for performing file I/O.

Each login shell maintains a description of the environment as a table of pointers to strings.

A global shell environment pointer called environ is maintained by UNIX kernel and it can be
used by a process to access its own table

The shell does not directly use this table, but it creates a child process and calls exec() system call
to execute a command program that uses the table inherited from the shell parent. Children
inherit the entire execution environment from the parent.

Some examples of the environment variables are the USER, LOGNAME, HOME, PATH, PS1, PS2,
TERM MAIL, etc.

The HOME Variable


It specifies an associated directory with every user in a UNIX system. If the HOME variable for the
user Sita contains /usr/sita/stores, every time Sita logs in, she is taken to the directory stores.
The variable HOME is referenced the same way:
$ echo ${HOME}

The PATH Variable


contains a list of all full path-names (separated by a colon) of directories that are to be searched
for an executable program. For example, the command
$PATH=.:/usr/bin:/bin specifies directories to be searched for any executable file or a command
file (current directory, /usr/bin and /bin, in that order).

The PS1 Variable


The system prompt may be changed by setting the value of this variable to the desired prompt:
$ PS1=“Hello>”
Hello> #can be changed only at the UNIX command line, not within a shell script.

The PS2 Variable: prompt string for continued command line (default ‘> ‘).

The LOGNAME Variable


contains user’s login name. Its contents cannot be changed by the user, but can be displayed:
echo “${LOGNAME}”

The TERM Variable


Names the kind of terminal you are using; setting it helps to manage your screen more
effectively, say,
$ TERM=vt100

The PWD Variable


The current working directory can be displayed:
echo “${PWD}”
In fact the whole environment table can be displayed.

$IFS: String of characters which are used as word separators in command line ( space, tab,
newline chars).

The MAIL Variable


Names the standard file where your mail is kept

The .profile File


Some of above variables like HOME and LOGNAME are set automatically each time the user logs
in. The others, however, have to be set. The .profile is used for this purpose as it is executed as
soon the user logs in. A sample .profile file would look like:
PATH=.:/bin:/usr/bin
export HOME PATH PS1 MAIL

The UNIX Process:

A process in UNIX is a program in execution with definite life-time and well-defined


hierarchy.

 The context of a process is a snapshot of its current run-time environment that


consists of the current state of the processor registers and
 User program image - Process execution on user data structure that represents an
application and is manipulated by its own functions (user mode execution).
 System image - Process execution on system’s data structures that represent the
resources (memory, files, devices) and supported by the kernel routines.
Depending upon resource requirement and availability, process’s states are
controlled by executing the kernel routines accessed through system calls (system
mode execution).
 The kernel maintains a process table to manage all processes. The two data
structures per process are the user structure and the process structure.
 The kernel process is the first (root) process that comes into existence when the
system is booted. Its process_id and group_id are both 0.
 In establishing a multi-programming environment, the kernel process creates the
init process with its process_id of 1 and group_id, 0, showing that process 1 is part
of process 0. The init process creates and manages terminal processes for active
terminals in the system.
 At the time of creation, each terminal process belongs to process group 0 and is
made to execute a program called getty. Once each terminal process (now called a
getty process) establishes communication with the associated terminal, it displays
a login message and waits for a user to input a user name (Process group of the init
process).
 When a user starts a dialogue, the getty process receives the user name and leaves
the task of validating the user’s password to another program called login. The
same terminal process is now called a login process.
 The login process, after validating the user’s password, calls a command line
program, the login shell to run in the same process. The same terminal process is
now called a shell process.
 Each shell process now establishes a new process group and becomes ready to
process the user commands. A shell process is the initiating process by which each
terminal maintains the user session.
 While interpreting a command, the shell creates an execution thread and then
assigns the requested command program to this new process.
 Both the shell and the new process proceed independently in separate execution
threads. The parent shell process normally waits until child process completes its
execution.

A process require certain resources to accomplish its intended task, they are
>> CPU time
>> System memory
>> Disk Files
>> I/O devices.

Process control blocks (PCB)


The OS must know all the information about a specific process in order to manage and control it.
The OS maintains a table (an array of structures), called the process table, with one entry per
process. These entries are called process control blocks (PCB) - also known as task control
block.This entry contains information about the process' state, its program counter, stack
pointer,memory allocation, the status of its open files, its accounting and scheduling
information, and everything else about the process that must be saved when the process is
switched from ready to running or blocked state so that it can be restarted later with the status
it left off.

Process state:- The state may be new, ready, running, waiting, halted, and so on.
Program counter:- The counter indicates the address of the next instruction to be
executed for this process.
CPU registers:- The registers vary in number and type, depending on the computer
architecture. They include accumulators, index registers, stack pointers, and general-purpose
registers, plus any condition-code information.
CPU-scheduling information:- This information includes a process priority, pointers to scheduling
queues, and any other scheduling parameters.
Memory-management information:- This information may include such information as the value
of the base and limit registers, the page tables, or the segment tables, depending on the
memory system used by the OS.
Accounting information:- This information includes the amount of CPU and real time used, time
limits, account numbers, job or process numbers, and so on.
I/O status information:- This information includes the list of I/O devices allocated to the process,
a list of open files, and so on.

Process state:
A process may present in one of the following state.
New:- The process is being created or just created.
Ready:- The process is waiting to be assigned/allocated to a processor time.
Running:- Instructions of the process are being executed by processor.
Waiting:- The process is waiting for some other event/process to execute.
Terminated:- The process has finished its execution.

ps command:
The ps (i.e., process status) command is used to provide information about the currently running
processes in the system. When ps is used without any option four columns of information
labeled PID,TTY, TIME and CMD for at least two processes, the shell and ps will be visible.
PID :- The process are identified by a 5 digit number known as PID (Process Identification
Number).
TTY :- Is the name of the console or terminal that the user logged into (Stands for terminal type
now
but originally stood for teletype).
TIME :- Is the amount of CPU time in minutes and seconds that the process has been running.
CMD :- is the name of the command that launched the process.

The information that ps -aux provides about each process is :


>> The user of the process,
>> PID of the process,
>> Percentage of CPU used by the process,
>> Percentage of memory used by the process,
>> VSZ (virtual size in kilobytes),
>> RSS (real memory size or resident set size in 1024 byte units),
>> STAT (the process state code, explained later),
>> Starting time of the process,
>> Length of time the process has been active
>> The command that initiated the process.

Types of process
Parent and Child Process:
A process can initiate a sub process, which is a called a child process, the initiating process is
referred to as its parent.The child processes, in turn create other child processes forming a tree
of processes
( which can be displayed using ps command with –forest option)
Orphan Process:
When a child process is killed, parent process gets the notification via a signal. Parent then, can
continue other task. However if the parent process is killed before, its child, is called an orphan
process.
Zombie Process:
When a process finished its execution and exit status not received by the parent ( or parent did
not read the exit status till now), the process state becomes zombie.
The process is dead (not to be scheduled for further execution) but cannot becompletely
removed from process table, until it has been determined that exit status is no longer needed.

Daemon Process
Some programs are not designed to be run with continuous user input and disconnect from the
terminal when task completed. For example, a web server responds to web requests, rather than
user input. Mail servers are another example of this type of application. These types of programs
are known as daemons.

Starting a Process:
Process can be started in two ways:
In Foreground: By default every process starts in foreground, ie. Gets the input from keyboard
and sends the output in monitor. But in this case till the process completes its execution no
other process can be started in foreground.
In Background: To take the advantage multiprocessing environment, a process can be started in
background, so that other process can be started in the foreground without waiting for the
previous process to complete execution.
A process can be started in background by adding ampersand(&) after it.

[390119 @ INGNRILPORCL] $ $ ls | wc -l > file1 &

Switching process from foreground to background:


A process running in foreground can be send to background using the following steps:
>> Press + Z to suspend the job
>> bg command puts the job in background
>> nohup unattaches the job from the terminal*

$ cat > file


File content...
+z
[1] + Stopped
$ bg
[1] + cat > file &

Switching process from background to foreground


Process running in the background can be taken into foreground using the following steps:
>> Find the job id of the process by the command jobs
>> Use fg %to get the job to foreground

Stopping/Killing a Process
A process dies(terminates ) automatically when it completes the job it intended to.
A process can be killed abnormally with the command kill.
>> Use the ps command to find out the process-id for the process
>> Use the command kill to terminate it
>> Use the command kill -9 to kill a process forcefully.
Option settings for background process
While a process is executing/running , if the owner tries to log off the process ,process will get
killed. Sometimes a job or command takes a long time to complete and it is required the job to
be completed without interruption. This situation can be handled in two different ways:
>> Does not allow the use to log off
>> Continue execution in background even after the user logged off

This can be achieved using command nohup

PERIODICAL EXECUTION OF JOBS

Sometimes it may be required to execute certain Job or task on specific time. It may be not
possible for any user to start or execute the job physically on that particular time due to other
business requirements. Again if similar task need to be executed periodically on a regular
interval, then the task becomes very hectic. Unix internally has solutions to handle this type of
situations using utilities such as cron and at.

Crontab
The word 'crontab' in UNIX stand for chron(chronograph) table, or time table for Unix system .
The crontab is a special table where it is possible to specify commands and time or interval to
execute the command once or repeatedly. Commands can be any executable programs, for
example, a script can be written to take regular backup of all the required tables in database. A
crontab instruction can be created to execute the script on the specified time. The cron daemon
reads the table and executes the commands at the times specified.

Crontab Command Syntax

Syntax
Components of Crontab
The basic format of a crontab schedule consists of 6 fields, placed on a single line and separated
by spaces, formatted as follows:
The various fields and their utility are explained in the image below:
Example:

To executes an incremental backup shell script "takeBackup.sh" available in your


home directory at 11:00 on every day the cron entry will be as below:

00 11 * * * ~/takeBackup.sh

To schedule the same job for every minute, the cron entry will be as follows:

* * * * * ~/takeBackup.sh

Вам также может понравиться