Automation - James - Bond Portal - Bond University

subduedlockMechanics

Nov 5, 2013 (3 years and 9 months ago)

71 views

Automation
Alexander Zangerl az@bond.edu.au
Bond University
Automation
1/20
don’t do things twice by hand!
￿
repetition of boring,tedious tasks does not increase quality
￿
likely to forget things
￿
likely to (ab)use command history (on the wrong files...)
Automation
2/20
cron
￿
runs commands periodically
￿
complex time specifications possible
￿
output is mailed back to a user
￿
predominant implementation:Vixie Cron
Automation
3/20
cron file format
￿
called crontab
￿
format:min hours day month weekday shell-command
*:for all
*/4:every 4th
0,3,5:at these specific times
￿
example:0 0 1 * * echo ”change this thing”
Automation
4/20
administration of cron
￿
frontend to user crontabs is called crontab
￿
crontab -l,crontab -e,other options dangerous
￿
user crontabs must be edited with crontab!
Automation
5/20
cron internals
￿
user crontabs in/var/spool/cron/crontabs (Debian)
￿
system crontabs:one extra field,the user to run as
￿
system-wide crontab in/etc/crontab
￿
Debian:extensive setup with run-parts and cron.daily,weekly
￿
cron mails back when output generated
￿
do check PATHs and environment for programs
￿
frequent problem:% starts comment!
Automation
6/20
cron-related tricks
￿
how to get output *only* in case of problems?
￿
problem:
￿
many programs can’t be quietened to the right degree
￿
e.g.:
￿
verbose,normal-but-still-too-much,dead quiet
￿
solution:
￿
( export TMP=‘mktemp‘;whatever...>$TMP 2>&1\
￿
|| cat $TMP && rm -f $TMP;)
Automation
7/20
at
￿
run stuff at specified time
￿
runs things only once
￿
relative time-specs available (eg.now+5minutes,teatime,...)
Automation
8/20
at administration
￿
”at timespec”,then interactively list commands to exec
￿
atq lists the queued jobs
￿
atrm removes queued job
￿
batch like at,but only runs stuff if load-level permits
Automation
9/20
at internals
￿
queued data stored similar to cron
￿
some UNIXes do have separate atd (Debian)
￿
others run some atrunner from cron every minute
￿
important to check what env (PATH) your commands are
given
Automation
10/20
shell scripting tricks
￿
get output of something:‘somecommand‘
￿
group with interpolation:”something”
￿
group without interpolation:’something’
￿
iterate over tokens:for i in...;do...;done
￿
cut to split stuff into columns
Automation
11/20
more shell scripting
￿
unix program exit code:0 good,everything else:bad
￿
test/[] is your friend!
￿
conditionals with || and &&
￿
pipelines can save your day
￿
xargs can do so,too
￿
find -mtime
￿
grep -v,grep -q
￿
diff,patch
Automation
12/20
sed – stream editor
￿
noninteractive,stream-oriented editor.
￿
interprets commandset,performs the actions on the input
￿
typical usage:
editing a file automatically
repetitive edits of multiply files
conversion programs
Automation
13/20
awk
￿
pattern matching program/programming language
￿
text file processor
￿
extensive pattern matching capabilities
￿
can deal with variable length records (cut:fixed length)
Automation
14/20
expect
￿
programmed dialog with interactive programs
￿
some programs do not accept input on STDIN,require TTY
￿
example:passwd
￿
solution:make PTY-pair and have expect control one side
￿
other side looks like a human to the program
￿
expect deals with matching and action-reaction patterns
￿
problem:synchronisation via output match
Automation
15/20
make
￿
generalized hierarchical organizer
￿
for operations on file objects
￿
originally intended for building programming projects
￿
dependencies are file-oriented
￿
actions are taken when dependent files are too old
￿
fine for updating stuff automatically whenever data is newer
Automation
16/20
make rules
￿
declarative language,”Makefile”
￿
rules specifying dependencies and how to update them
￿
example:
target:dependencies
[tab] rules
￿
important:target(file) must start on first character of line
￿
rule must start with tab!
￿
first target in makefile is default
Automation
17/20
scp
￿
copy files to a remote location,securely
￿
if coupled with public-key-based authentication:automatable
￿
but:copies full files,no synchronization features
￿
also key must be unprotected or ssh-agent available
Automation
18/20
rsync
￿
synchronize files over the network
￿
transfers only the differences of files
￿
very efficient,works for binaries too
￿
uses rcp or ssh as transport backend
￿
THE method of choice for automated data transfers
Automation
19/20
unison
￿
file synchronization tool,but more advanced than rsync
￿
uses the rsync protocol
￿
but allows merging of conflicting versions,
￿
detects which version should be copied where
￿
interactive,so not fully automated
￿
available for Windows and Unix
Automation
20/20