12/5/2012

COMP 175 System Administration Ad i i t ti and Security

Class Objectives
Install OS, utilities, applications Maintain it patches, backups Manage files, directories, permissions a o do u a o sys o s Learn how the documentation system works Make it secure Monitor it

% cat "food in cans" cat: can't open food in cans

You will learn how to: Install, Configure, and Manage Securely A server providing Internal and Internet Services without looking like a n00b or an id10t

Challenges
Difficulty with teaching system administration is that the industry and technology changes much faster than the typical textbook and coursework t d i i t t is th t Diffi lt with b i Difficulty ith being a system administrator i that the industry and technology changes fast. The scope of knowledge required is also increasing.
Components Operating system Application software Networks & Network services Storage Virtualization Security

Sysadmin Duties
Analyzing system logs and identifying potential issues Introducing and integrating new technologies Performing routine audits of systems and software Performing regular backups (and testing the backups) e o g egu a bac ups (a d test g t e bac ups) Applying OS updates, patches, and configuration changes Installing and configuring new hardware and software Adding, removing, or updating user account information Resetting passwords Answering technical queries, dealing with frustrated users Responsibility for security Documenting the system configuration Troubleshooting any reported problems. System performance tuning. Ensuring network infrastructure is up (monitoring)
4

It Begins

It Begins

12/5/2012

Ownership
If you dont own it you cant count on it What is not measured, cannot be managed p g , p y / Evil has to sleep at night, stupidity is 24/7 Bad documentation is worse than none at all Inspect everything Things rarely happen during 8-5 Manage management I ffi i t i Insufficient paranoia Benthic technology Maintenance
5

History
Ken Thompson Bell - UNIX Dennis Ritchie Bell UNIX - C g Brian Kernighan Bell - C Bill Joy vi System V (commercial) - BSD Forks POSIX standards for API, shell, tool interfaces

It Begins

Linux Version 2.6

Compliant with IEEE POSIX standard All features of a modern UNIX OS
Virtual memory, virtual file system, lightweight processes, Unix signals, SVR4 (System V, Release 4) inter-process communications, support for S i ti t f Symmetric M lti t i Multiprocessors (SMP)

Epoch Fail
Differences between UNIX variants will be most apparent to the system administrator. The Unix epoch is 00:00:00 UTC on 1 January 1970 = 0 On 32-bit systems this count (or the integral portion thereof) overflows on: Tue Jan 19 03:14:08 2038 (UTC) y At that time the date suddenly becomes: Fri Dec 13 20:45:52 1901

Monolithic kernel
Most commercial UNIX variants are monolithic Exceptions: Apple MAC OS X and GNU Hurd OS (both derived from CMU Mach OS) follow microkernel approach

Support for modules on demand automatic load/unload

O l SVR4.2 and Solaris kernel have similar feature Only SVR4 2 d S l i k lh i il f t

Kernel threading
Kernel organized as a set of kernel threads Only Solaris and SVR4.2 are organized like this
It Begins 7

12/5/2012

Operating System
An Operating System controls (manages) hardware and software. It provides support for peripherals such as keyboard, mouse, display, disk drives, Software applications use the OS to communicate with peripherals. The OS typically manages (starts, stops, pauses, etc) applications. In simple terms, an operating system is a manager.

Parts of the UNIX OS

The Kernel The Shell and Graphical User Interface(s) GUI y Built-in System Utilities Application Software and Utilities
4 3 2 1

Operating System Overview

Operating System Overview

10

Operating System Concepts

The monolithic design is older and uses a single binary image to provide the resource management and hardware interface functions of th core l l f the lithi the layer. S Some examples of th monolithic design are Linux and Solaris. A micro-kernel design uses a very small task management component and a suite of modules for all other resource management functions. NT, 2000, Windows NT Windows 2000 Windows XP and Mac OS X are examples of micro-kernel designs. CLI vs GUI
Operating System Overview 11

X Window System
X window: program that draws windows on the screen under most GUI-based versions of UNIX. X windows consists of 2 distinct parts the X server and 1 or more X clients The X window server runs on the machine to which the monitor is connected. The server controls the display directly, and is p p / p y , responsible for all input/output via the keyboard, mouse or display. Window Manager KDE, TWM, FVWM, Fluxbox Desktop Manager KDE, GNOME, CDE
Operating System Overview 12

12/5/2012

Help Commands
RTFM - SysAdmin acronym for: Read The Man page ( ) (or Read The Manual) Similar to: GIYF ("Google is your friend") LMGTFY ("let me google that for you"). man man (to learn how to use it better)
User space

File Systems
File System Architecture
User Applications GNU C Library

13

Kernel space s

14

File Systems
Can you fill these out? Try it till you can.

Block & Character Devices

Block devices move data to/from that occur in blocks (such as disk sectors) Supports buffering and random access behavior (is not required to read blocks sequentially, but can access any block at any time). Block devices include hard drives, CD-ROMs, RAM disks. Character devices differ in that they do not have a physically-addressable media. Character devices include serial ports and tape devices, in d i i l d i l t dt d i i which data is streamed character by character.

3 4 5

2 6 7 10 8 9

15

16

12/5/2012

File Systems
Journaling keeps track of the major steps taken during last file sessions ACID (atomicity, consistency, isolation, durability) a set of properties that guarantee database transactions are processed reliably. If one part of transaction fails, transaction fails ext4 journaling successor to ext2 swap used to support virtual memory p pp y NTFS MS file system - ACLs and journaling FAT 32 USB sticks HFS+ - OS X - journaling
17

File Systems
/dev/sda Pri. Master Disk /dev/sda1 1st partition /dev/sda2 2nd partition

fsdsk Inconsistent State

Drive Multiple partitions 1 or more platters 1 + Heads per side 1 ring around track Block 512 byte Cylinder

Fdisk Cfdisk
18

File Systems
What are the four parts?
A B C D

RAID

19

RAID (Redundant Array of Inexpensive Disks Software software drivers Hardware special controller p Fake BIOS + multi-channel controller RAID-systems can use several interfaces, including SCSI, IDE, SATA or FC (fibre channel.) JBOD, Just a Bunch Of Disks no RAID level RAID is no substitute for Back-Up! Back Up! Logical Volume Manager arrays of drives Direct Attached Storage: a JBOD storage array embedded RAID
20

12/5/2012

RAID Levels
RAID 0: Striping - faster Data split into blocks that are written across all drives in array RAID 1: Mirroring Data stored twice by writing to both data disk(s) and a mirror disk(s) 2 disk minimum RAID 5: Parity Across Disks Datablocks are subdivided (striped) and written across multiple drives Parity information is spread across all the drives 3 disk minimum
21

Storage
NAS: Network attached file server appliance SAN: a separate Storage Attached Network g Storage Virtualization
Abstracting logical storage from physical storage

Now playing - Gigabyte Terabyte Petabyte Coming Attraction - Exabyte Zettabyte Yottabyte Information life-cycle life cycle Storage (media) life-span Legacy hardware available?

bit rot

22

/proc
/proc directory is a pseudo-filesystem /proc/swaps /proc/ide /p / /proc/network /proc/cpuinfo

Booting
Master Boot Record vs Superblock ext4 Volumes size up to 1 exabyte File sizes up to 16 terabytes LILO Linux Loader GRUB Relative vs absolute addressing A big-endian machine stores the most significant byte first, and a little endian machine stores the least significant little-endian byte first

On a UNIX system, everything is a file; if something is not a file, it is a process." file process

23

24

12/5/2012

Filesystem Classes
Filesystems can be categorized into classes: Image - special filesystem that presents the modules in the image and is always present Block - traditional filesystems that operate on block devices like hard disks and CD-ROM drives. Flash - Nonblock-oriented filesystems designed explicitly for the characteristics of flash memory devices. NOR devices - FFS3 NAND - ETFS. Network - provide network file access to the filesystems on remote hosts. NFS and CIFS (SMB) Virtual - resource manager that sits in front of other file systems
The File System 25

Access Permissions
r Indicates that a category of user can read file w Indicates that a category of user can write to file g y x Indicates that a category of user can execute file owner The owner of the file or application group The group owning the file or application others All users with access to the system chown chgrp chmod di t ~Y Your h home directory .. The parent directory . The current directory
26

The Shell
The shell is a command line interpreter file standard input (for reading) file standard output (for writing normal output) p ( g p ) file standard error (for writing error messages) Bourne shell sh C Shell csh Korn Shell ksh ash h Bash Bourne-Again shell - typical default shell Default prompt is $ (# for root user)
27

Shells
Shell programming First line starts with #! a hashbang shebang or sharp bang g g p g absolute path to shell

Set executable bits on the file with chmod, e.g.: $chmod +xshell_script

28

12/5/2012

Daemons
A computer program that runs in the background Usually initiated as background processes yp y Typically daemons have names that end with "d" syslogd - the system log daemon sshd - handles incoming SSH connections httpd web server daemon

Services
DHCP protocol and a mechanism to hand out configuration parameters to hosts IP addresses + gateways + time server + DNS Leases Automatic allocation DHCP assigns a permanent IP address to a client Dynamic allocation DHCP assigns an IP address to a client for a limited period of time (or until the client explicitly relinquishes p ( p y q the address) Manual allocation IP address is assigned by network administrator, DHCP used to convey the assigned address to the client
29 30

Cron enables users to schedule jobs inetd - 1 daemon many apps

DHCP
DHCP Protocol DORA Discover Offer Request Acknowledge And A d Release, Decline, NAck

DHCP Process
Lease duration Client holds IP when not connected Clients retire /D t b h ld h t t IPs S Servers/Databases should have constant IP Analyze The Network Sufficient addresses available? Performance? Servers? Redundancy is available Failover is available DHCP must be up before clients use a UPS
31 DHCP 32

DHCP

12/5/2012

DHCP Terms
Scopes and exclusions A pool of IP addresses that can be assigned to clients Reservations IP addresses can be reserved for specific computers using MAC addresses Leases Clients no longer own their own IP address and instead lease one from a DHCP server The lease has a time limit but can be renewed
DHCP 33

DNS
DNS provides the translation function between the two Internet namespaces: The domain name hierarchy The Internet Protocol (IP) address space i.e. Resolves domain names to IP addresses Abstracts URL's and email addresses

34

DNS
First implementation - 1983 BIND written - 1984 ISC BIND is the most widely used DNS software y On Unix systems it is the de facto standard Domain name consists of dot delimited labels FQDN Fully.Qualified.Domain.Name. Right-most label is the Top Level Domain (TLD)
.aero .asia .biz .coop .gov gov .info .int .jobs .mil .museum .name .mobi .pro .travel .xxx
35 Domain Name System

TLD's
air-transport industry Asia-Pacific region companies business cooperatives limited to US government entities informational international organizations job advertisements US military museums individuals mobile devices professions tourism and travel sexually explicit
36

Domain Name System

12/5/2012

DNS Lookup Example

DNS Lookup Example

www.cs.pacific.edu www cs pacific edu Client

root & edu DNS server

www.cs.pacific.edu www cs pacific edu Client

root & edu DNS server

ns1.pacific.edu 2 DNS server DNS local resolver 3 Iterated query: Contacted server replies with name of server to contact I dont know this name, but ask this server
Domain Name System 37

2 DNS local resolver 3

ns1.pacific.edu DNS server

cs.pacific.edu DNS server

cs.pacific.edu DNS server

Recursive query: Contacted server replies with resolved answer I know this name, here is answer. I'm not authoritative though".
Domain Name System 38

Lookup using cached DNS server

root & edu DNS server

DNS records
DNS: distributed db storing resource records (RR)
RR format: (name, value, type, ttl)

www.cs.pacific.edu Client

Type=A Type A
ns1.pacific.edu DNS server Local DNS recursive resolver

name is hostname value is IP address

Type=CNAME
name is alias name for some canonical (the real) name www.ibm.com is really: seast.backup2.ibm.com l l value is canonical name

cs.pacific.edu DNS server

Type=NS
name is domain (e.g. foo.com) value is hostname of authoritative name server for this domain
Domain Name System

Caching servers improve efficiency, reduce DNS traffic, and reduce latency by storing query results for a period of time (TTL).
Domain Name System 39

There are ~35 record types

40

10

12/5/2012

DNS Records
Type=MX
name is Mail eXchange record value is hostname of l i h t f the Mail Transfer Agent for the domain

DNS Amplification Attack

Type=PTR
Pointer Record value is pointer to a canonical name i l

Type=SOA
start of authority record value is the primary il name server, email contact, domain serial number, and several zone TTL timers
41 Domain Name System 42

Type=AAAA
IPv6 address record value is 128-bit IPv6 address

Domain Name System

Log Lifecycle
Keyword: Retention Throwing away log files Not recommend Security problems - accounting data and log files provide important evidence of break-ins Performance problems - alert you to hardware and software problems general, In general keep one or two months Compromises may not be obvious Subtle performance issues may not be obvious
System Logging 43

Circular Rotation
Rotating log files Keep backup files that are 1 day old, 2 days old logfile, logfile.0, logfile.1 , logfile.2, logfile.6 g , g , g g , g - or logfile, logfile.1, logfile.2 , logfile.3, logfile.7 Which is more logical? clear? Each day rename the files to push older data toward the end of the chain

System Logging

Log rotation

44

11

12/5/2012

Linux Example
/var/log dns_queries.log g faillog maillog messages /var/log/apache access_log l error_log /var/log/samba

Configuration File
Severity Level Level emerg (panic) alert crit err warning notice info debug Approximate meaning Panic situation Urgent situation Critical condition Other error conditions Warning messages Unusual things Informational messages For debugging

Unlike facilities, which have no relationship to each other, priorities are hierarchical. Wildcards are * and none. A priority may be preceded by either or both of the modifiers = and !
46

0 1 2 3 4 5 6 7

System Logging

45

System Logging

System Logging

Backup In Context
Backups are routine task Test that they work Worst case is a failure while b ki up backing Backups also part of disaster recovery plan strategic Large data losses often the result of lost backups

A 2011 study of small businesses: 81% consider data to be their most valuable asset 57% lack a disaster recovery plan for data 40-60% never re-open after a disaster (FEMA) System/hardware failure accounts for 68% of data loss Human error accounts for 32% of data loss
System Logging 47 Backing Up Data 48

12

12/5/2012

Designing a Backup Strategy

Backup plan Written document that outlines: When and how files are backed up H How fil are stored files t d How files are restored Backup plan questions What files should be backed up? Who will back up files? Where are files located? How should backups be performed? Must you be able to restore data within a specific period of time? (SLA)
Backing Up Data 49

Designing a Backup Strategy

Determining value of data Spend more $$ to protect the integrity of expensive data Opportunity cost Determine when to back up data Data changes frequently in most organizations
Constitutes daily work of users within organization User data Log files E-mail archives

Backing Up Data

50

Designing a Backup Strategy

Backup level Defines how much data is backed up Backup operation at given backup level stores p p g p all data that has changed since last backup of previous level Levels Level 0, full backup 1, Level 1 weekly differential backup Level 2, daily differential backup

Designing a Backup Strategy

Full backup Also called epoch backup Everything on system is backed up y g y p Differential backup stores only files that changed since full backup Incremental backup stores files that changed since most recent incremental backup or p differential backup Separation of data for different backup options
51 Backing Up Data 52

Backing Up Data

13

12/5/2012

Breaktime
Eyes tired? Take a break! YouTube time.
http://www.youtube.com/watch?v=XPWZym4mA6E Fill in the box at the top of the final exam as shown below. It worth two extra points.
Q: Everybody needs an: A: Education

vi
Bill Joy wrote vi as a grad student at UCB There are two modes in vi ( y) Command mode (esc key) : esc :w write esc :q quit Input (Insert) mode a (append) after current character t) before current character t h t i (i (insert) b f dd delete line / pattern string search Viva la vi
53 54

Ok back to the notes Actually use the review notes? Two free points!

Budgeting
What is a budget? A planning tool What were going to do next year g g y
And why we need to do it

Scripting
Most popular languages are based on ALGOL hierarchical in structure environment nesting g control structure nesting dynamic arrays reserved words user defined data types Common modern languages on web servers C d l b PHP Perl Ruby Python

How much its going to cost to do it What to expect in future years Hardware Software People Services Consumables Slush fund Two kinds of expenditures: Capital Expensed
55

56

14

12/5/2012

Firewalls
Firewalls implement a security policy Packet filters stateless simplest type pp y p ,p Apply rules to packets based on address, port Packets either dropped or rejected Stateful filters act based on state e.g. connection response to a handshake Examine header field, content NAT PAT NAT, Proxy server
57

SSH
Secure Shell (SSH) - replaces telnet Key exchange yp Encrypts traffic Default port 22

58

Samba
Server Message Block (SMB) message format Common Internet File System (CIFS) protocol (MS) CIFS/SMB used for printer and file sharing fi ll Bl k ports 137 139, 445 at firewall Block 137, 139 nbtstat -n netstat for SMB net commands Samba implements SMB/CIFS on Unix Can be a Windows server or client Provides 4 basic Common Internet File System services: File and print services Authentication and Authorization Name resolution Service announcement (browsing)
59

Scripting Languages
Most popular languages are based on ALGOL
ALGOrithmic Language (1950's) hierarchical in structure environment nesting control structure nesting dynamic arrays reserved words user defined data types

Perl PHP Ruby Python Java

60

15

12/5/2012

PHP
Server-side scripting language PHP code embedded into HTML source p y Interpreted by web server Always sanitize inputs Done wrong vulnerable to: Command injection i j ti SQL injection Session hijacking

Virtual Memory
Extends amount of Physical memory Linux: Memory = swap + RAM Swap space should be 2X Physical Memory Swapping Moving pages to and from memory Page block (unit) of RAM (4k) Thrashing excessive swapping Solution: More memory, better coding y, g

61

62

Processes
Process is the executing program code May include resources such as open files, pending signals, internal kernal data, processor state, address space, global variables data Multiple, independent processes all running Many per user Background daemons running wake up look for work do work sleep Kernel manages processes time sliced

Processes
Processes are named by their PIDs Parent can fork(spawn) a child process Unacknowledged terminated children zombies g InterProcess Communication via pipes Processes can communicate via software interrupts & after command background

63

64

16

12/5/2012

Signals
1. SIGHUP 9. SIGKILL hangup kill kill HUP (pid) kill -9 (pid)

Tuning
Virtual memory statistics provide information useful for application and system tuning and problem solving. Tuning important in Multi-Processor Architectures

ps Proces Status top terminal screen display prstat Solaris t t Viewing i t l i f ti vmstat Vi i virtual memory information Windows tasklist (cmd line)

Package Management
Slackware "Dependency management is left up to the sysadmin"

65

66

Package Management
Old school compile from source New School package managers Packages lessen library problems g yp Dependency issues apt-get Advanced packaging tool (Debian) Various GUI's for apt-get apt-get Synaptic rpm yum YaST Zypp

Web Server
1989 Tim Berners-Lee wrote world's first web server (it ran on NeXTSTEP Apple bought basis for OS X) Market leader today is Apache Attributes of a good web server tt butes o eb se e 1. Correctness conforms to spec 2. Reliability high uptime 3. Scalability works when needed the most 4. Stability grace under pressure 5. Speed no noticeable degradation g g Advantage of awareness and some understanding of: Hardware characteristics and features Operating System internal workings Application configurations with respect to OS, hardware Can make significant performance difference
67 68

17

12/5/2012

Performance Considerations
RoundRobin DNS Load balancing g p y Geographical redundancy Application Considerations Read/Understand the Instructions/Documentation Test in a test environment L and Monitor when in production h i d ti Log d M it Watch for advisories, Patch as needed Avoid being exploited its part of the job
69

Avoid

UNDETECTED COMPROMISE

DATA DISCLOSURE

DEFACEMENTS

70

NTP
Network Time Protocol UTC (Coordinated Universal Tim) Civil time measured on an atomic time scale Kept within 0.9 seconds of astronomical time 09 Atomic clocks accuracy 1 sec in 6M years Computer clocks keep poor time Clock drift - Gap over time between clocks Clock skew difference between two clocks Track adjustments and apply continuously Disciplining the clock

NTP
Assumes no machine has an accurate time source Obtain average from participating computers Synchronizes all clocks to a fault-tolerant average y g Sanity checks ran on the timestamps/servers Arranged in a strata 1st stratum: machines connected directly to accurate time source 2nd stratum: machines synchronized from 1st stratum machines and so on Stratum 0 is a physical clock, never a computer
72

71

18

12/5/2012

NTP
Longest running, continuously operating, ubiquitously available protocol in the Internet (since 1979) 10-15 millisecond accuracy for a typical network
VM
App

Virtualization
VM
App

VM

App

Guest OS

Guest OS

...

Guest OS

Virtual Machine Monitor (VMM) Platform HW

Memory CPU I/O Devices

Virtual Machine capable of virtualizing all hardware resources, processors, memory, storage, and peripherals Virtual Machine Monitor (VMM) provides virtual machine abstraction Also referred to as hypervisor
73 74

VMM Hypervisor (Here a b (H bare-metal hypervisor) lh i )

Virtualization
VM
App

Virtualization
App

VM

App

VM

Can you label the parts? Try it.

Guest OS

Guest OS

...

Guest OS 1 2 3 1 3 2

...

1 3

Virtual Machine Monitor (VMM)

Host Operating System Platform HW

Memory CPU I/O Devices 6 7

Running the Virtualization layer on top of a host OS Be able to label the components.
75

76

19

12/5/2012

In Closing
Review this slide deck If a slide here is not clear go back to the source slide desk and any notes you have Bolded/Blue highlight terms know them See you at the final exam - 9:00

77

20