BeginnersShellScripting for BatchJobs

EvanBollig GeoffreyWomeldorff
and

Beforewebegin...
Everyonepleasevisitthispageforexample scriptsandgrabacribsheetfromthefront
http://www.scs.fsu.edu/~bollig/TechSeries

Alsopleasesigninontheclipboardbeing passedaround

Outline
IntroductiontoShells
History Whatisthe*nixENV? WhatisShellScripting?

NoviceScripting
HelloWorld(HelloWorld.sh) Variables(HelloWorld2.sh) `...`,'...'and...(Example3.sh)

 IntermediateScripting

Outline(2)

FileandOutputControl(Example4.sh) Loops,ConditionalsandUtilizingAvailablePrograms (Example5.sh)

AdvancedScripting
SubstringsandCaseStatementsforMachineandArchitecture SpecificTasks(Example6.sh) RestartFiles,IntegersandAvoidingRedundantTasks (Example7.sh) CallingMatlabforBatchProcessing(Example8.sh)

IntroductiontoShells

Whatare*nixShells?
UnixwasthefirstOStoseparatethecommand interpreter(shell)fromtheoperatingsystem(kernel) Shellsabstractthecomplexitiesofsystemsand providestandardcommandsforallmachines Ashellhasbuiltincommands,plusallowsusaccess tootherUnixcommands
history,export,cd>builtin cp,ls,grep>Unixcommands(/bin,/usr/bin,etc.)

http://www.phys.ualberta.ca/~gingrich/research/shells/shells.html

AGeneralHistory
BourneShell(/bin/sh):
BellLaboratories,byStephenBourne Basisforallshells,butprovidedlimitedfunctionality

CShell(/bin/csh):
UCBerkley,byBillJoy(creatorofvi) ExpandedBourneShelltomimicClanguageandprovideinteractivity(in reality,itwastoobuggytouseforscripts) TENEXcsh(/bin/tcsh)waslaterdevelopedtogetridofbugsbutUnix manufacturersstuckwithcsh

http://www.softlab.ntua.gr/facilities/documentation/unix/shelldiff.html#3

History(2)
KornShell(/bin/ksh):
AT&T,byDavidKorn Hybridofshandcshusingshlanguagewithcshinteractivity Althoughitwasthebestofbothworlds,itwasstillfromAT&T=>ksh wasnotfree.

BourneAgainShell(/bin/bash):
GNU'sPOSIXcompatibleBourneShell EquivalenttoKornbutfreeimplementation PerfectfitinLinuxwhichresultedinmasspopularity

rc,zshandothersarestillemerging...

UnderstandingtheENV
TheENVistheUnixenvironment
Key=valuepairsdescribecurrentconfiguration Everyrunningapplication,user,etcthatinteractswiththe operatingsystemhasanenvironment Aprocessinheritsitsparent'senvironmentunlesstheparent overrides/specifiesotherproperties. Users'initialENVisprovidedbytheshellandbasedon /etc/profile.d/*,/etc/*shrcand~/.*shrcfiles. Changestochild'senvironmentonlyaffectchild'schildren andarelostwhenthesubtreeexits.

ControllingtheEnv
Shellsallowustocontroltheenvironmentwithbuiltin commands;Unixcommandscanalsobeused: %>env
Listsyourcurrentenvironment

%>export<key>=<value>
Setanewkeyorupdateanexistingone NOTE:thisisequivalenttosetenv<key><value>intheCshell family

%>echo$<key>
Printcurrentvalueofvariable

UnderstandingShellScripts
Shellscriptsarefilesthatcontaincommandsasyou wouldrunthemonthecommandprompt. Scriptscommonlyusevariables,conditionalsandloops.
Theseareallavailableonthecommandprompt

Whatmakesascriptsospecial? Ithastheexecutablebitset(chmod+x<script>) Itexecutesuntilitencountersanerror(interpreted) Scriptscanuseanyavailableexecutables;noneedto worryaboutlibraries,headersorAPIs

WhyScript?
Quickanddirtysolutions
Mostlyyouusepreexistingbinaries Changestoscripthaveimmediateeffects noneedtowrite/(re)compilecode

Interprogramcommunicationisperformedthrough standardfileI/Opipes
Noworriesaboutcommunicationstandards ShellScriptsuseshellsthatarestandardonall*nixplatforms NOTE:onlytheshellsarestandard.Thisisnotalwaystrue forthecommandsascriptcontains.

NoviceShellScripting

HelloWorld
Example1:HelloWorld.sh Keyfeatures:
Shbang(#!/bin/sh)
Everyshellscriptmuststartwithashbangonthefirst line.Thistellstheexecutingshellwhattypeofshellshould interpretit(i.e.sh,bash,csh,perl,python,tcl,etc.)

Comments
Commentsbeginwiththehash(#)andproceedtotheend oftheline

Variables
Example2:HelloWorld2.sh;Hello.sh KeyFeatures:
LocalandGlobalVariables
<key>=<value>isalocalvariablethatisnotpassed tochildren export<key>=<value>isaglobalvariablethatis passedtochildrenbutisnotvisibletoparentofthis script

`...`,'...'and... Example3:Example3.csh KeyFeatures:

Useofdifferentshell(seecshinshbang) `...`:executeprogramperformingsubstitution onvariables '...':stringswithoutvariablesubstitution ...:stringswithvariablesubstitution

IntermediateShellScripting

FileandOutputControl
Example4:Example4.sh;HandsFreeCharmm.sh KeyFeatures:
cat<<EOF...
Printeverythingbetween<<EOFandthenextoccurrenceof EOFtostdout. NOTE:the>example4.outtotherightof<<EOFredirects stdouttoafile

Redirectingbothstdoutandstderr
Use([command]>[stdoutFile])>[stderrFile]tostoreoutput separately

Use[command]&>[outFile]tostoretogether

Loops,ConditionalsandUtilizingAvailable Programs
Example5:Example5.sh KeyFeatures:
Forloopexecutesonlistoffilenames Conditional(ifthenelse)testsforalockfiletoavoidextra work WetakeadvantageofstandardUnixcommandslikemkdir, pwd,grepandechobutalsousenonstandard Imagemagicktoperformbatchimageconversion

AdvancedShellScripting

SubstringsandCaseStatementsforMachine andArchitectureSpecificTasks
Example6:Example6.sh KeyFeatures:
Casestatementtocomparemachine'sname(hostname) Casestatementtocomparearchitectureofmachine Substringsfromvariablesusing${var_name#*/}and ${var_name%/*}

RestartFiles,IntegersandAvoiding RedundantTasks
Example7:Example7.sh KeyFeatures:
Restartfileinformsscriptwhattasksithasalready completed Integersrequireuniquehandling(wecannotcompare likestrings) LocatesALLjpegsinALLsubdirectoriesandrebuilds subdirsinoutputfolder(importanttokeepbatchjobs organized)

CallingMatlabasforBatchProcessing
Example8:Example8.sh;testMatlab.m; mandrill_cvt.m KeyFeatures:
Permachinetasks
differentcompressionforeachcluster:Class*,Prism*, Hallway*,andVislabmachines

ExecutesMatlabbehindthesceneswithacomplex clusteringalgorithmfordatacompression(easilycoded inMatlabandeasytovisualizeresults)

Conclusion
Younowhaveexampleshellscriptstogetstarted. UsetheresourcesonthecribsheetandontheSCS TechSupportTwikitohelpyouwriteyourownscripts. Uselogging,locksandrestartfilestohelpaccelerate batchjobs. Remember:don'treinventthewheel!Ifacommandexists anditfunctionscorrectly,don'twasteyourtimerewriting itfromscratch.Ifithasbugsthenitsanotherstory...

SpecialThanks...
toTSGfortheexcusetoplaywithshellscripts toGordonErlebacherfortheextratimeallotment awayfromWATT toSCSCustodialformakinglifeinterestingduringthe weehoursofthemorning

Andofcourse!Thankyouallforattending...