Cut-and-paste as a scripting language
I'd say one, perhaps controversial, technique that has been very
successful for me in computer system administration are scripts that
are not intended to be run with a shell or other single tool but as a
series of things to cut-and-paste. I frequently use such things in
situations like:
- Drafting a proper scripts to be run from Puppet, Chef, Ansible,
cron, etc.
- Massaging existing systems from one state into another before
Puppet, Chef, or Ansible takes over management. This includes
bootstrapping.
- Changing state in things not under Puppet, Chef, Ansible, etc,
control, because, eg it's a database not an OS.
- The code is to be used as a runbook for what should be done in the
case of an error.
In many cases the cut-and-paste script is something that should become
a proper script if it is going to be done often enough, but see point
one. A number of systems that need to be brought into a consistent
state but for whatever reason are starting from a diverse set of
existing states might need a script with a lot of branching. The
cut-and-paste script has a human running it and human that can perform
error recovery and new branch configurations with relative ease.
To point two: in some cases there will be enough systems to perform
this on that some script is required, but the test to know which state
should apply is a complicated one to script and it's much simpler to
let a human decide the steps needed after which the test becomes
simple and automation can take over.
And for point three: always you will have some aspect of the system
that is supposed to be above the level of the automation but for which
a degree of control is sometimes needed.
Lastly point four: a runbook that provides as exact as possible set of
steps to follow allows for more error-free recovery from issues when
they do arise. Sometimes this set of steps can be shown to be reliable
enough to create scripts (point one again) that perform autorecovery.
I think at this point it becomes clear that the cut-and-paste script
is a useful developmental tool for creating robust automation. At
least, I frequently find it to be so.
Some examples of situations I've used cut-and-paste scripts:
- I first developed this method managing a fleet of web servers
in the early '00s. The site and the config were all in source code
control, but triggering syncs and Apache config reload were manual
actions. A loop like
for host in $(< hostlistfile); do ssh $host "cd /webroot && p4 sync"; done
would work, but it wouldn't be nearly as close to simultaneous as
opening sixteen xterms each sshed in to a web server (easy to
locally script) and then pasting the commands as fast as possible in
each. Later (much later) that company started to use Chef, where the
knife command could do a good job of replacing such stuff.
- Using a web browser to connect to "console" of a newly installed
system, using
xdotool to type the commands to bootstrap networking
onto the system. That "console" was some weird javascript app that
didn't accept "paste", hence getting creative with xdotool to
emulate typing in it. That network had no DHCP and needed a static IP on
the correct VLAN before it could connect. I broke down the commands
into several xdotool commands for two reasons, (a) to keep the
command lines from getting unreasonably long (ie personal taste),
(b) to not "typeahead" when starting programs like a file editor,
and (c) to not have to encode which static IP to use, instead just
getting right to that point, entering it by hand then continuing with
the script. Finally the script ended with rebooting, and then I could
take over with ansible, the config management tool of choice there.
-
Filling out a number of web forms where an API should be used, but
there is resistance to making the API usable. Specifically, of late,
that has been managing "silences" during system updates in Prometheus
Alertmanager. Due to the login being behind Okta, command line tools
can't authenticate. There is a proposed fix for this, but it hasn't
been prioritized yet. In the meantime, I'll just open
vi in an xterm
and put field values to use on separate lines for quick triple-click
copying. Typically I'll have two files open one with the things that
are the same for every "new silence" and one for the hostnames changing
between them.
One thing that has helped with all of this is the standard X11
paradigm of "select is copy" and "mouse middle button is paste".
I can double click on a word in one window, move the mouse to another
window and paste it, with no keyboard. Or triple click for line, and
then paste it in as many windows as I want with just a single click
each. It becomes the opposite of hands never leave keyboard, where
the script run is completely done with hand never leaves mouse (at
least until the script needs an edit). This cut and paste paradigm never caught on,
and it makes me less productive on Macs and Windows. Macs are in some
ways the worst, because it's Unix but it's not X11: almost, but not
quite. (And to add to the pain, Mac keyboard shortcuts are nothing like xterm
ones.)
Of course, if you do need to type some part of the cut-and-paste script
learning and using keyboard shortcuts for that, particularly paste, are
the way to go. Consider this simple script:
for host in $(< hostlist ) ; do
# need to set EDITOR enviornment variable after sudo
ssh -t $host "sudo env EDITOR=ex crontab -e"
done
# paste to ex:
# :g /day$/ s, \* /, 1-5 /, | x
One can copy that global command, which finds lines ending in "day"
and edits the * before the / to be 1-5. For a crontab like:
MAILTO=cron-list@example.com
3 9 * * * /usr/local/bin/start-day
10 2 * * * /usr/local/bin/run-backup
This will change run every day at 9:03 am start-day program to
only run on weekdays: 3 9 * * 1-5 /usr/local/bin/start-day and
save the file. The for loop will log in to each host in the list
of hosts, possibly prompting for an ssh password (depending on how
you have that set up) and probably prompting for a sudo password
(depending on how you have that set up). It would be quite reasonable
to run the loop, enter passwords by hand, then hit <shift-insert>
to use the default xterm paste primary keybinding to paste the script
whereupon cron is updated, you are logged out, and move on to next
host, so you end up just typing
password<enter><shift-insert>
password<enter><shift-insert>
password<enter><shift-insert> ....
Some tricks of the trade that I've picked up over the years of doing
this:
- Many things that don't immediately look cut-and-pastable can be
made so by rethinking them. File edits with
ex or ed are easier
to script than vi; xdotool can be used to control things in
windows.
- Whole lines are easier to copy than fragments, triple-click will
select a line.
- Learn exactly how to include or not include trailing new lines in
your copies, it can make a lot of difference.
- Use variables like you would in a regular script loop for things
that will vary from run to run, even if each run is on a separate
host. This lets you separate variable setting pastes from command
run pastes.
- Setting variables with shell commands instead of manually makes
for a smoother experience.
grep, head, and tail, are good
for selecting lines from output. cut, sed, and grep -o are
good for selecting part of a line from output:
host=$(hostname -f)
homeuse=$(df /home | tail -1 | grep -o '[0-9]*%' | cut -f 1 -d %)
- Some shell line break methods are more fragile than others.
Consider:
# Pasted as a group, both of these run, even if you only want the
# second to run if the first succeeds.
/bin/false
/bin/true
# Pasted as a group, the second will run only if the first succeeds
/bin/true &&
/bin/false
# The backslash will escape the following whitespace character.
# If you have a trailing space, that's the whitespace that will
# be escaped, not the newline.
[ -f "$somefile" ] &&
head "$somefile" \
| next command
# Written like this, trailing whitespace is ignored, the pipe
# alone is enough to keep the shell looking for more to do.
[ -f "$somefile" ] &&
head "$somefile" |
next command
And remember, cut-and-paste scripts are a good way to write a real
script, slowly dealing with all the error cases to handle and best
ways to make the code universal. Use them to compose your startup
scripts, your cron jobs, your configuration management tooling, your
runbooks, and coding your self-healing systems.
qz thoughts