Unix Operating System Tools and commands

bartolomeo.montrucchio@polito.it maurizio.rebaudengo@polito.it

Politecnico di Torino Dip. Automatica e Informatica 2010/2011

Operating systems - Tools

Original copyright from Giorgio Di Natale, Stefano Di Carlo and Fabio Maino

Outline

Some commands

find grep tar, gzip, gunzip sort sed awk

Filters

Operating systems - Tools

File search

find <dir> [-opt] some options:

-name pattern Warning: use single quotes if you are using regular expressions -type [b c d l]

Operating systems - Tools

Advanced use of find

It is possible to execute a command on all found files specifying: -exec command \; In command, use \{} for the current file example: find . -name core -exec rm \{} \;

Operating systems - Tools

Advanced use of find

Search expressions can contain logical conditions Logical operations:

AND: conditions must be listed one after the other OR: use -o NOT: use !

Operating systems - Tools

Advanced use of find

Use ( ) to build complex expressions Example: find . \( -name core -o -size 10b \) find . \( -name test -o ! \( -size 10c \) \) \ is to protect ( from the shell expansion; 10c means 10 byte, while 10b means 10 blocks

Operating systems - Tools

grep

In order to search a string in a set of files, it is possible to use: grep [-options] pattern files

Operating systems - Tools

grep

Options:

-c counts lines which contain the pattern -i case unsensitive -l shows only names of the files containing the pattern (usually also the line is shown) -n shows the line number -v shows only the lines not containing the pattern
Operating systems - Tools 8

Regular expressions with grep

Patterns in grep can be strings or regular expressions Some characters have a special meaning (if there is not a preceding \)

. ^ $

any character beginning of a line end of a line

Operating systems - Tools

Regular expressions with grep

* repetition (zero or more times) + repetition (one or more times) [] any character among those in parentheses [^ ] a character excluding those in parentheses \< beginning of the word \> end of the word

Operating systems - Tools

10

Recursive search in a diretcory tree

In order to do a recursive search in a directory tree: find . -name * -exec grep pattern \{} \; Please note that you should use and not ` (using ` launches the shell)

Operating systems - Tools

11

tar

tar [option] files Stores and extracts files in a tarfile

Operating systems - Tools

12

tar - archive creation

Options to create a tarfile:

-c -f file -v

creates a new tarfile name of the tarfile verbose

Example:

tar -cvf /tmp/maino.tar /home/maino

Operating systems - Tools

13

tar - archive extraction

Options to extract from a tarfile:

-x -t -f file -v

extracts files from tarfile tests tarfile name of the tarfile verbose

Examples:

tar -tvf /tmp/maino.tar tar -xvf /tmp/maino.tar

Operating systems - Tools 14

gzip

gzip [opt] file compresses a file options:

-1 -9

fastest max compression

Operating systems - Tools

15

gunzip

gunzip file Decompresses a file

Operating systems - Tools

16

Filters

A filter is a program receiving data from stdin and producing results on stdout Filters are very useful with I/O redirection and with pipes Examples:

more, less head, tail

Operating systems - Tools

17

Data sorting

sort [-options] [file ...] options:

-b ignores beginning blanks -d considers only blanks and alphanumeric characters -f case unsensitive

Operating systems - Tools

18

Data sorting

sort [-options] [file ...]

-o file writes results on file (default is on stdout) -r reverse ordering -tcar field separator Eg. sort k2.2,2.3 filename orders filename on a field based on characters from 2 to 3 of the second field of the file

Operating systems - Tools

19

sed - Stream text EDitor

It is a filter for processing the file as given by script options sed [-n] script [file ] sed [-n][-e script][-f script_file][file ]

Filters stdin as specified in script -n does not repeat stdin on stdout; prints only what is requested from script

Operating systems - Tools

20

sed - Stream text EDitor

Script syntax:

[address[,address]] function [args] address: line number or a regular expression function: command on the pattern match args: args of function

Operating systems - Tools

21

sed functions

p prints current line d deletes current line q quits y/orig/subs/ modifies orig in subs

Operating systems - Tools

22

sed functions

s/regexp/replace/flags replaces patterns satisfying regexp with replace flags:

num g p

replaces only num occurrences replaces all the occurrences prints the line if it has replaced something

Operating systems - Tools

23

sed: examples

sed 1,3 d filename (deletes from line 1 to 3) sed 3,$ d filename (deletes from line 3 to the end of file) sed -n /^pluto/ p filename (prints only if starts with pluto, because of -n) sed -f sedfile filename

Operating systems - Tools

24

{ s/^/Begin:/ sed: esempi s/$/ -- End/ } sed 1,3 d filename /\/\*.*\*\// d

1,1

sed 3,$ d filename sed -n /^pluto/ p filename sed -f sedfile filename

Operating systems - Tools

25

{ s/^/Begin:/ sed: esempi s/$/ -- End/ } sed 1,3 d filename /\/\*.*\*\// d

1,1

sed 3,$ d filename sed -n /^pluto/ p filename sed -f sedfile filename

One Two Three ABC /* Remark */

Operating systems - Tools

26

{ [only for the first line] s/^/Begin:/ sed: esempi s/$/ -- End/ } sed 1,3 d filename /\/\*.*\*\// d [deletes remarks]

1,1

sed 3,$ d filename sed -n /^pippo/ p filename sed -f sedfile filename

One Two Three ABC /* Remarks */

Begin: One Two Three --End ABC

Operating systems - Tools 27

Awk

awk was invented in 1977 from

A. V. Aho P. J. Weinberger B. W. Kernighan

It is a computer language based on string pattern recognition

Operating systems - Tools

28

Basic principle

For each line of the open file, patterns are searched When a match is found, an action is done Syntax is similar to C language

Operating systems - Tools

29

Input file

Input file is organized in records (file line) A record is organized in fields (words of the line)

Operating systems - Tools

30

Input file

Variables:

RS: FS: $0: $1: $n:

record separator field separator (blank) whole record first field n-th field

Operating systems - Tools

31

Execution

It is possible to run awk both from command line and from a script Command line:

awk command < inputFile > outputFile awk -f scriptFile < inputFile > outputFile

Script:

Operating systems - Tools

32

Command structure

Each awk command is composed by a pattern and by an action pattern {action;}

Decides when action is executed

Can be made by one or more instructions. It is executed only if pattern is true

Operating systems - Tools 33

Command structure

If there is not a pattern, action is always executed If there is not an action, the current line is printed

Operating systems - Tools

34

Examples

awk $1==Ciao < inputFile

prints all lines in which the first word is Ciao prints the first field of each line of the file

awk {print $1;} < inputFile

Operating systems - Tools

35

Patterns

Patterns can be:

regular expressions / expr / comparison operators $1 == Ciao operators for intervals ( cond1, cond2 )

Operating systems - Tools

36

Regular expressions

\ ^ $ . [abc] [a-z] [^abc] a, b, c

escape sequence beginning of the line end of the line one character one of the characters sequence of characters all characters with the exception of
Operating systems - Tools 37

Regular expressions

one|two can be one or two * 0 or more times of the preceding character + 1 or more times of the preceding character ? [AB]? can be A or B or the null string

Operating systems - Tools

38

Regular expressions

()

combines regular expressions Logi(c|cal) is satisfied by: - Logic - Logical

Operating systems - Tools

39

Regular expressions Examples

/^((may)|(MAY)|(May))$/ { print May; } /^[Tt]itle.*/ { print \nNew title.; }

Operating systems - Tools

40

Comparison operators

== < > <= >= != ~ !~

equal less than greater than less or equal greater or equal not equal equal as a regular expression not equal as a regular expression
Operating systems - Tools

41

Comparison operators

&& || !

logical AND logical OR logical NOT

Operating systems - Tools

42

Comparison operators Examples

$1 == Bob { print Bob stuff; } $1 !~ /[Mm]aggio/ { print Not may; } ($1 == Bob) && ($2 ~ /[mM]*xy?RR$/) { print First field is Bob. Second is wrong; }

Operating systems - Tools

43

Interval operators

cond1, cond2

Pattern is true only when first condition is true and remains true until second condition becomes true It works only with nawk o gawk

Operating systems - Tools

44

Interval operators

$1==1, $1==CIAO { print $2; }

Test ciao 1 number one 2 number two CIAO End 3 number three
Operating systems - Tools

number number End

45

Special patterns

BEGIN: run before opening input file END: run after having read all input file

Operating systems - Tools

46

Example

BEGIN { FS=:; } $1 ~ /[0-9]/ { print $3; }

Test:ciao 1:number:one 2:number:two CIAO:End 3:number:three

Operating systems - Tools

one two three

47

Actions

Variables Strings Arrays Operators If Loop Input Functions Shell interaction

Operating systems - Tools 48

Variables

Variables can be:

A field, with a $ ($0, $n) Predefined User defined (declaration is not requested)

Operating systems - Tools

49

Predefined variables

ENVIRON

Environment variables (it is an array). E.g.: ENVIRON[PATH] FS field separator (in input) IGNORECASE 0 means case sensitive, 1 no NF number of fields NR number of records already read
Operating systems - Tools 50

Examples
3

print NF; print $NF;

AA BB CC

CC

print $NF;
Operating systems - Tools

$NF
51

String operators

Union:
Putting strings one after the other: awk '{x=Hello"; y= everybody"; print x,y;} < /etc/hosts Hello everybody Hello everybody Hello everybody (because /etc/hosts ha three lines)

Operating systems - Tools

52

String functions

sub (reg, string, target), gsub (reg, string, target)

substitutes in target the first occurrence (all with gsub) of the substring satisfying regular expression reg con string lenght of the string s gives position in string of the first substring satisfying regular expression reg
Operating systems - Tools 53

length (s)

match (string, reg)

String functions

printf (s, ), sprintf (s, )

as in C splits string into the elements of array vec given delimiter delim. Returns the number of elements in vec.

split (string, vec, delim)

substr (string, position, len) tolower (s), toupper (s)

Operating systems - Tools 54

Arrays

With numeric index:

v[5] i=3; v[i]=CIAO day[january] = 31; day[february] = 28;

With symbolic index :

Operating systems - Tools

55

Arrays

Example:

day[january] = 31; day[february] = 28; january in day ==> gives true delete day[march]

Functions for arrays with symbolic index:

Operating systems - Tools

56

Multidimensional arrays

With numerical index:

V[5,3] V[abc, def] V[1999, january]

With symbolic index:

With mixed index:

Operating systems - Tools

57

Operators

x+y, x-y, x*y, x/y x^y, x%y ++x, x++, --x, x-sin(x), cos(x) rand( ) (random number among 0 and 1) systime( ) (seconds starting from January 1st 1970)
Operating systems - Tools 58

If

if (cond) { trueinstruction } else { falseinstruction } condition ? trueinstruction : falseinstruction;

Operating systems - Tools

59

Loop

do { instruction } while (condition) for (init; cond; op) { instruction } for (i in array) { instruction }

Operating systems - Tools

60

Loop (cont)

while (condition) { instruction }

Operating systems - Tools

61

Advance input

exit

is like the end of the file (if END pattern exists, it is run) reads a new line and copies it in $0. Returns 1 if the reading has been successful.

getline

Operating systems - Tools

62

Functions

Work only with nawk or gawk function myFunc (parameters) { ... return x; }

Operating systems - Tools

63

Shell interaction

system (shellCommand);

runs a shell and executes specified command; e.g. system(ls la) print Ciao > filename.txt; print myVar | more;

using redirection

Operating systems - Tools

64

Example

Bookmark file:
+|category url1|description url2|description ...

+|University http://www.polito.it|Politecnico di Torino http://www.unito.it|Universit di Torino +|Music http://www.mp3.com|Mp3 web site

Operating systems - Tools 65

Example
<HTML><HEAD><TITLE>My bookmarks</TITLE></HEAD> <BODY> <H1>University</H1><BR><UL> <LI> <A HREF="http://www.polito.it"> Politecnico di Torino</A></LI> <LI> <A HREF="http://www.unito.it"> Universit di Torino</A></LI> </UL> <H1>Music</H1><BR><UL> <LI> <A HREF="http://www.mp3.com"> Mp3 web site</A></LI> </UL> </BODY> Operating systems - Tools 66 </HTML>

Header
BEGIN { FS = |; firstTime = 1; printf (<HTML><HEAD><TITLE>); printf (My bookmarks</TITLE></HEAD>); printf (\n<BODY>\n); }
Operating systems - Tools 67

Title
$1 == + { if (firstTime == 0) { print </UL>; } else { firstTime = 0; } print <H1> $2 </H1><BR><UL>; }
Operating systems - Tools 68

URL and final part

$1 != + { print <LI><A HREF=\ $1 \ > $2 </A></LI>; } END { print </UL></BODY></HTML> }
Operating systems - Tools 69