QZ qz thoughts
a blog from Eli the Bearded
Tag search results for unix Page 1 of 4

Cut-and-paste as a scripting language

I'd say one, perhaps controversial, technique that has been very successful for me in computer system administration are scripts that are not intended to be run with a shell or other single tool but as a series of things to cut-and-paste. I frequently use such things in situations like:

  1. Drafting a proper scripts to be run from Puppet, Chef, Ansible, cron, etc.
  2. Massaging existing systems from one state into another before Puppet, Chef, or Ansible takes over management. This includes bootstrapping.
  3. Changing state in things not under Puppet, Chef, Ansible, etc, control, because, eg it's a database not an OS.
  4. The code is to be used as a runbook for what should be done in the case of an error.

In many cases the cut-and-paste script is something that should become a proper script if it is going to be done often enough, but see point one. A number of systems that need to be brought into a consistent state but for whatever reason are starting from a diverse set of existing states might need a script with a lot of branching. The cut-and-paste script has a human running it and human that can perform error recovery and new branch configurations with relative ease.

To point two: in some cases there will be enough systems to perform this on that some script is required, but the test to know which state should apply is a complicated one to script and it's much simpler to let a human decide the steps needed after which the test becomes simple and automation can take over.

And for point three: always you will have some aspect of the system that is supposed to be above the level of the automation but for which a degree of control is sometimes needed.

Lastly point four: a runbook that provides as exact as possible set of steps to follow allows for more error-free recovery from issues when they do arise. Sometimes this set of steps can be shown to be reliable enough to create scripts (point one again) that perform autorecovery.

I think at this point it becomes clear that the cut-and-paste script is a useful developmental tool for creating robust automation. At least, I frequently find it to be so.

Some examples of situations I've used cut-and-paste scripts:

  • I first developed this method managing a fleet of web servers in the early '00s. The site and the config were all in source code control, but triggering syncs and Apache config reload were manual actions. A loop like for host in $(< hostlistfile); do ssh $host "cd /webroot && p4 sync"; done would work, but it wouldn't be nearly as close to simultaneous as opening sixteen xterms each sshed in to a web server (easy to locally script) and then pasting the commands as fast as possible in each. Later (much later) that company started to use Chef, where the knife command could do a good job of replacing such stuff.
  • Using a web browser to connect to "console" of a newly installed system, using xdotool to type the commands to bootstrap networking onto the system. That "console" was some weird javascript app that didn't accept "paste", hence getting creative with xdotool to emulate typing in it. That network had no DHCP and needed a static IP on the correct VLAN before it could connect. I broke down the commands into several xdotool commands for two reasons, (a) to keep the command lines from getting unreasonably long (ie personal taste), (b) to not "typeahead" when starting programs like a file editor, and (c) to not have to encode which static IP to use, instead just getting right to that point, entering it by hand then continuing with the script. Finally the script ended with rebooting, and then I could take over with ansible, the config management tool of choice there.
  • Filling out a number of web forms where an API should be used, but there is resistance to making the API usable. Specifically, of late, that has been managing "silences" during system updates in Prometheus Alertmanager. Due to the login being behind Okta, command line tools can't authenticate. There is a proposed fix for this, but it hasn't been prioritized yet. In the meantime, I'll just open vi in an xterm and put field values to use on separate lines for quick triple-click copying. Typically I'll have two files open one with the things that are the same for every "new silence" and one for the hostnames changing between them.

One thing that has helped with all of this is the standard X11 paradigm of "select is copy" and "mouse middle button is paste". I can double click on a word in one window, move the mouse to another window and paste it, with no keyboard. Or triple click for line, and then paste it in as many windows as I want with just a single click each. It becomes the opposite of hands never leave keyboard, where the script run is completely done with hand never leaves mouse (at least until the script needs an edit). This cut and paste paradigm never caught on, and it makes me less productive on Macs and Windows. Macs are in some ways the worst, because it's Unix but it's not X11: almost, but not quite. (And to add to the pain, Mac keyboard shortcuts are nothing like xterm ones.)

Of course, if you do need to type some part of the cut-and-paste script learning and using keyboard shortcuts for that, particularly paste, are the way to go. Consider this simple script:

for host in $(< hostlist ) ; do
        # need to set EDITOR enviornment variable after sudo
	ssh -t $host "sudo env EDITOR=ex crontab -e"
# paste to ex:
#	:g /day$/ s, \* /, 1-5 /, | x 

One can copy that global command, which finds lines ending in "day" and edits the * before the / to be 1-5. For a crontab like:

3 9 * * * /usr/local/bin/start-day
10 2 * * * /usr/local/bin/run-backup

This will change run every day at 9:03 am start-day program to only run on weekdays: 3 9 * * 1-5 /usr/local/bin/start-day and save the file. The for loop will log in to each host in the list of hosts, possibly prompting for an ssh password (depending on how you have that set up) and probably prompting for a sudo password (depending on how you have that set up). It would be quite reasonable to run the loop, enter passwords by hand, then hit <shift-insert> to use the default xterm paste primary keybinding to paste the script whereupon cron is updated, you are logged out, and move on to next host, so you end up just typing password<enter><shift-insert> password<enter><shift-insert> password<enter><shift-insert> ....

Some tricks of the trade that I've picked up over the years of doing this:

  • Many things that don't immediately look cut-and-pastable can be made so by rethinking them. File edits with ex or ed are easier to script than vi; xdotool can be used to control things in windows.
  • Whole lines are easier to copy than fragments, triple-click will select a line.
  • Learn exactly how to include or not include trailing new lines in your copies, it can make a lot of difference.
  • Use variables like you would in a regular script loop for things that will vary from run to run, even if each run is on a separate host. This lets you separate variable setting pastes from command run pastes.
  • Setting variables with shell commands instead of manually makes for a smoother experience. grep, head, and tail, are good for selecting lines from output. cut, sed, and grep -o are good for selecting part of a line from output:
       host=$(hostname -f)
       homeuse=$(df /home | tail -1 | grep -o '[0-9]*%' | cut -f 1 -d %) 
  • Some shell line break methods are more fragile than others. Consider:
    # Pasted as a group, both of these run, even if you only want the
    # second to run if the first succeeds.
    # Pasted as a group, the second will run only if the first succeeds
    /bin/true &&
    # The backslash will escape the following whitespace character.
    # If you have a trailing space, that's the whitespace that will
    # be escaped, not the newline.
    [ -f "$somefile" ] && 
    	head "$somefile" \ 
    	| next command
    # Written like this, trailing whitespace is ignored, the pipe
    # alone is enough to keep the shell looking for more to do.
    [ -f "$somefile" ] && 
    	head "$somefile" | 
    	next command

And remember, cut-and-paste scripts are a good way to write a real script, slowly dealing with all the error cases to handle and best ways to make the code universal. Use them to compose your startup scripts, your cron jobs, your configuration management tooling, your runbooks, and coding your self-healing systems.

About Post Ordering

One thing that is tricky with Blosxom is that posts are (unless changed by plugin) ordered strictly by modification time. At first blush, this seems very reasonable. But eventually one finds problems.

A naïve backup and restore can totally thrash file modification times, and leave everything in a random unpredicatable order.

# don't backup Blosxom like this, restore will be broken
cp blog/* /mnt/backup-disk/blog/

Then there are subtler things. In my previous use of this blog, I replaced the sorting plugin to show most recent posts on first page, and alphabetal posts on all category pages. That suited my "bookmark blog" usage very well. It doesn't suit my current usage, and I disabled that sort when I restarted.

But when I restarted I also moved the hostname. And I edited the old posts that had references to the hostname. I edited the old posts.... I lost the original time stamps on those two posts, and I don't know when they are from. I probably have the timestamps in a tar backup somewhere, but that's not handy since it's old. (Panix has backups, too, but not in a format easy to get an old timesstamp from.) I "solved" that by using touch to change the dates back the same same month of my last posts. It's not right, but it's a lot more right than February 2020.

Enter my quick fix for now, a tool to save time stamps in a way that is very easy to restore from. I call it SAVE-DATES and it's a wrapper around a tool I have not used for anything else in many years: super ls (sls). Quoting the README I wrote for it:

In 1989 this program was released to the world in the form of being posted to the Usenet group comp.sources.unix. It was in volume 18 of the archives, which probably exist out on some dusty ftp server still. (But a quick search did not find such an ftp server in October 2017. The links run cold at the Internet Systems Consortium's sources.isc.org.)

Many of the features of sls were new at the time, but are less radical now. And the growth of interpreted languages for system admin use. Languages like Perl (described as a "replacement" for awk and sed, when posted to c.s.u, and archived in volume 13) and Python provide native access to stat(2). And now there's a straight-out stat(1) in Gnu coreutils.

I found this program in the archives somewhere around 1993 and immediately took a liking to it, and patched it many times for my own needs. Today I barely use it, sls having fallen to the wayside due to several factors: non-standard interface (the printf style output control isn't really printf(3)), the rampant buffer overrun problems in the code, more powerful all-in-one output features available with Perl.

The heart of sls that I still really like is the easy way I can turn a simple directory listing into usable code. And that's how my quick fix works. SAVE-DATES uses sls to create a shell script of touch commands that will restore dates. The whole thing can be run, or lines greped out to selectively (and more speedily) be used.

#              [[CC]YY]MMDDhhmm[.ss]
export SLS_DATEFMT=%Y%m%d%H%M.%S
# use sls to make shell commands to reset date/time on files
slsoutstyle='touch -t %m %n'
find "$datadir" -type f -exec sls -p "$slsoutstyle" {} + > "$savefile"

The environment variable makes the date format the same as the date in format touch wants. Then I have the output format of sls include the fixed strings needed for touch to run, plus the modification time (%m) and name (%n). Note that the names are raw, so it relies on me picking "safe" names for post files, but that's my habit anyway. The output looks like this:

touch -t 202002222054.00 /net/u/3/e/eli/public_html/qz/data/blog/feb-2020-a-reboot.txt
touch -t 202002230325.45 /net/u/3/e/eli/public_html/qz/data/blog/how-its-built.txt
touch -t 202002240050.59 /net/u/3/e/eli/public_html/qz/data/blog/haircuts.txt
touch -t 202002250217.49 /net/u/3/e/eli/public_html/qz/data/blog/first-patch.txt
touch -t 202002270203.01 /net/u/3/e/eli/public_html/qz/data/blog/very-impressive-phone.txt

Date restore is easy-peasy now.

Unix Text Processing

The 1987 Dougherty and O'Reilly book on the text processing tools of Unix has been released under a Creative Commons Attribution license. This site is coordinating the effort to recreate the book in groff source from page scans.

dead link: http://home.alltel.net/kollar/utp/