Earliest Viable Poop

There are two dogs in this household, with very different personalities. Dog walks highlight one of the biggest contrasts.

In this end-of-dog-walk shot, I've leashed the dogs to a pillar while I put the garbage bins away. Willie stands in the center of the sidewalk, staring into the distance. Hazel is trying to get back into the house.

Willie is super skilled at mental maps. He probably cannot get himself lost (although he has been known to run off). When we traveled, in pre-covid times, Willie could always be counted on to know when we were back "home" even if it was at night, somewhere he's been inside of only once for a few minutes, and approaching from a different direction than he had visited it before.

Hazel, two years after adoption, now seems to almost never go up the neighbors stairs. Hazel likes a walk from time to time. She is always happy to go on the morning walk (okay, not happy on rainy days), although she sometimes needs some help to start moving towards the door. But come evening, she'll pointedly run away from the leashes.

Willie is usually — rainy days are an exception — happy to walk for as long as I'd like to walk, and I'm quite willing to walk a long time. Our evening walks are often around half a mile, and sometimes longer than a mile. Morning walks, such as in the above photo are just around the block.

Which brings me to "Earliest Viable Poop".

In corporate nomenclature, the "minimum viable product" is the least advanced version of something a company thinks they can get money for. Companies like Apple don't generally release the minimum, but you can be sure a lot of apps in the Apple App Store are minimum viable. The idea behind such is "release early and use the revenue to pay for the development of more features."

Hazel clearly prefers to poop outside, but if the outside is unavailable or too unpleasant, she'll just use that bathmat next to the toilet. (That's what that room is for, right?) That's the Earliest Viable Poop for Hazel, the poop before the walk even starts. If she was releasing smart phone apps, they'd do one thing, and just barely do it all.

Willie, he will make it clear when he wants to go out and when he needs to to out. And once out, if only at the "wants" stage, he uses his mental map to make a good guess about the walk length. Even if it's not a route he's walked before, he is good at sensing if it is now the return trip. When he guesses that it is past half-way (or on a route he knows well, more than three-quarters) then he starts to really pay attention to the good spots. Even in rain, where he has what we call "efficient" walks, the earliest viable poop is a half block away (which is also about the distance of the last viable poop on a regular return trip). He takes his time to find the right spot, and has standards about the whole business. His threshold for minimum product would not be solely based on the minimum a low discerning customer would accept, but the minimum someone with real choice would consider.

I like to think about these things on long meditative walks with Willie. I know there is a place for the barely complete solution, but remember it's on par with pooping on the bathroom floor. Technology and scat have more in common than people think. A bunch of stuff goes in, gets digested, and then a release is made to the world, sometimes with a lot of care about how it lands, sometimes not.

My Alarm Clock

When I was a kid, maybe twelve, I got a German made wind-up alarm clock, the type with the two bells on top and separate springs for the clock and the alarm. That was my "daily driver" of an alarm for a long time. After I married, my wife hated it because the ticking was "loud" when she was trying to get to sleep. When one of the springs finally broke, probably about 2005ish, I switched to using my cellphone as an alarm. Some Nokia brick. Eventually I switched to a Audiovox flip "feature" phone.

The Audiovox was Nokia quality indestructable. (But it was in many ways a terrible phone.) I used it as a phone for years and then when I finally upgraded, I continued to use the Audiovox as an alarm clock for some more years. I replaced it only in 2016.

On the Audiovox the alarm mode had three alarms that could be set, and then enabled or disabled. I found that three different wake-up times covered 90+% of my usage: wake up on a day when the kids have school, wake up on a day when I have work but the kids don't have school, and weekend. Then there would be 5 to 10% of the time when I'd need a special time for something.

So when it came to replacing it, I wanted an alarm clock that would give me three different standard times and one special, and I wanted an interface that could let me set a week of them at once. I turned to a "pocketCHIP" (hereafter Pocketchip) tiny portable computer.

Pink buttons show alarms on, white ones show alarms off. The day of week button turns off all alarms for that day. Time of day is shown on the current day of week. Code available from my github page, it's Perl with Tk for the X11 UI.

The Chip was a $9 computer, the "Pocket" part was an attachment to add a keyboard, touchscreen, battery pack, headers for GPIO etc, and a small hacking space. Overall it cost about $60, which was (and is) a steal compared to what a Raspberry Pi outfited as a laptop / tablet / netbook / other portable computer costs. As of 2020, you can still find new-old-stock on ebay for $50ish. Unlike a Pi, Chip was low-res: no HDMI option; and has built-in wifi and bluetooth and built-in storage. Now Pocketchip isn't great, it's got a take-you-back-to-the-late-1980s pixel count (480x272), a painfully awkward keyboard, no built-in speakers, and poor audio volume from the audio jack.

I modified mine adding a basic mono amplifier (PAM8230 from Adafruit) and a speaker attached to that. I also connected one of the GPIO pins to a standard 1/8th inch (2.5mm) audio jack, to use with an external button. There was ample room in the case for the amp, speaker, and extra jack. I did need to cut the plastic a bit to get get the proper parts sticking out.

The green board in the upper left is for the GPIO 2.5mm jack, the blue board near it is the add-on amp, there's an on-off switch for the amp in the far upper left, and the speaker is squeezed in on the bottom right. The larger black board is the Chip $9 computer, and the silver thing below it is the rechargable battery.

Then I built a button out of a wooden box and Cherry D42 switch with a lever that I pulled out of some device. The lever holds the top of the box slightly open, pushing the lid down closes the switch. Big easy to push button for sleepy alarm silencing.

When the alarm fires, a program plays a sound file (repeatedly up to 30 times or 60 seconds as configured) and listens to a GPIO pin. Every 1/100th second the GPIO pin is sampled. With ten high samples and ten low samples — to allow for a switch normally open or normally closed — it decides that's enough and kills the sound playback and quits.

This part was made entirely with reused stuff I had laying around.

I wrote the alarm software sometime in 2016 and have made only slight modifications to it since then. It works, and it works well. When I built myself a new bedside table last year, I designed it to have a place for the Pocket chip to hang, and suitable space for all the wires.

I made the lamp first, then the alarm button to fit under the lamp, then the table that this all lives on. The drawer holds pencils and bookmarks, the current night time reading is on the table, the next books to read on the first shelf, books done reading go on the shelf below that. All of this is from scavenged or left-over material.

As this setup gets long in the tooth, I'm starting to think about what I do next. In particular, I worry about the Chip failing and/or the battery becoming useless. I've been thinking it might make a good learn Arduino project. I'd prefer to spend less than $100 on all parts, have a system with at least the current capabilties, including normally plugged in but with at least an hour's backup power to ride out blackouts. And a screen that I can easily turn off for darkness. It's been fun browsing, but I don't have a parts list yet.

Grilled Pizza

I've cooked a lot of pizza over the years. These day's I'm using the Ken Forkish Same Day Straight Pizza Dough recipe, but I was using a simpler one of my own for a while:

• 3 1/2 to 4 cups flour
• 1 1/3 cups warm water
• 2 oz, plus some olive oil
• yeast
• dash of sugar
• 1 tsp salt (or to taste)

Mix the water, sugar, and yeast and set aside. Measure out 3 1/2 cups flour and mix in the salt. Pour in the oil and yeast water together, stir to a dough, and knead in more flour until it is not sticky, then knead some more. Pour some more oil in the bottom of a bowl, put the dough in and rotate to cover the dough surface with oil, cover with a damp cloth and put aside to rise. The oil, both in the dough and on the surface, and the damp cloth help a lot. Let rise a while, how long will depend on temperature, at least one hour but probably two, refrigerated it might take eight. Then divide in half and shape.

I developed that recipe after getting a pizza stone for Xmas in maybe 1997. The stone broke after three to five years and I got a better stone which is still going strong. Usually I preheat the oven as high as it will go and give the stone at least twenty minutes at that temperature to soak up heat. I generously use coarse corn meal (sometimes called "polenta") on the peel to keep the dough from sticking to the peel or the stone.

But today was not usual. Today it was very hot and I decided to try the barbecue grill method I've heard about. I have a propane BBQ with a cast iron grill that I cooked directly on. First off was an extra thorough cleaning of the grill, then preheating it. With a shaped but untopped round of pizza, put it directly on the metal grill and cook for two minutes. Then scrape it off, which was pretty easy. (I used a metal pizza peel, but spatulas would work.) Grill mark side up, top it, then return to the grill for five minutes. At this point the dough was cooked, but the cheese could have used a little more heat, so I finished it with a hand propane torch.

I have a trigger start propane torch that is intended to sit on top of a 14oz tank, Ace Trigger Start Torch, I've seen nearly the same used in commercial kitchens to cook meringue, so I don't have qualms about using it for food cooking. I just waved it back and forth until the cheese was nice and bubbly and it came out great.

I expect I'll be cooking pizza this way again.

Off the grill with one side cooked, ready to be topped. The underside is slightly sticky, so be careful sliding it off.

Back on the grill for some more cooking. I found gently lifting one side off the peel, pulling the peel out to the opposide of that, then dropping lifted part worked well. (I cooked it, both sides, with lid closed.

Bubbly and slightly browned is just right.

I made four pizzas, one completely consumed by the time this photo was taken. Two plain cheese, one pepperoni, and one pesto.

I also made, with leftover dough from the recipe, a BBQ focaccia (not pictured).

Cut-and-paste as a scripting language

I'd say one, perhaps controversial, technique that has been very successful for me in computer system administration are scripts that are not intended to be run with a shell or other single tool but as a series of things to cut-and-paste. I frequently use such things in situations like:

1. Drafting a proper scripts to be run from Puppet, Chef, Ansible, cron, etc.
2. Massaging existing systems from one state into another before Puppet, Chef, or Ansible takes over management. This includes bootstrapping.
3. Changing state in things not under Puppet, Chef, Ansible, etc, control, because, eg it's a database not an OS.
4. The code is to be used as a runbook for what should be done in the case of an error.

In many cases the cut-and-paste script is something that should become a proper script if it is going to be done often enough, but see point one. A number of systems that need to be brought into a consistent state but for whatever reason are starting from a diverse set of existing states might need a script with a lot of branching. The cut-and-paste script has a human running it and human that can perform error recovery and new branch configurations with relative ease.

To point two: in some cases there will be enough systems to perform this on that some script is required, but the test to know which state should apply is a complicated one to script and it's much simpler to let a human decide the steps needed after which the test becomes simple and automation can take over.

And for point three: always you will have some aspect of the system that is supposed to be above the level of the automation but for which a degree of control is sometimes needed.

Lastly point four: a runbook that provides as exact as possible set of steps to follow allows for more error-free recovery from issues when they do arise. Sometimes this set of steps can be shown to be reliable enough to create scripts (point one again) that perform autorecovery.

I think at this point it becomes clear that the cut-and-paste script is a useful developmental tool for creating robust automation. At least, I frequently find it to be so.

Some examples of situations I've used cut-and-paste scripts:

• I first developed this method managing a fleet of web servers in the early '00s. The site and the config were all in source code control, but triggering syncs and Apache config reload were manual actions. A loop like for host in $(< hostlistfile); do ssh $host "cd /webroot && p4 sync"; done would work, but it wouldn't be nearly as close to simultaneous as opening sixteen xterms each sshed in to a web server (easy to locally script) and then pasting the commands as fast as possible in each. Later (much later) that company started to use Chef, where the knife command could do a good job of replacing such stuff.
• Using a web browser to connect to "console" of a newly installed system, using xdotool to type the commands to bootstrap networking onto the system. That "console" was some weird javascript app that didn't accept "paste", hence getting creative with xdotool to emulate typing in it. That network had no DHCP and needed a static IP on the correct VLAN before it could connect. I broke down the commands into several xdotool commands for two reasons, (a) to keep the command lines from getting unreasonably long (ie personal taste), (b) to not "typeahead" when starting programs like a file editor, and (c) to not have to encode which static IP to use, instead just getting right to that point, entering it by hand then continuing with the script. Finally the script ended with rebooting, and then I could take over with ansible, the config management tool of choice there.
• Filling out a number of web forms where an API should be used, but there is resistance to making the API usable. Specifically, of late, that has been managing "silences" during system updates in Prometheus Alertmanager. Due to the login being behind Okta, command line tools can't authenticate. There is a proposed fix for this, but it hasn't been prioritized yet. In the meantime, I'll just open vi in an xterm and put field values to use on separate lines for quick triple-click copying. Typically I'll have two files open one with the things that are the same for every "new silence" and one for the hostnames changing between them.

One thing that has helped with all of this is the standard X11 paradigm of "select is copy" and "mouse middle button is paste". I can double click on a word in one window, move the mouse to another window and paste it, with no keyboard. Or triple click for line, and then paste it in as many windows as I want with just a single click each. It becomes the opposite of hands never leave keyboard, where the script run is completely done with hand never leaves mouse (at least until the script needs an edit). This cut and paste paradigm never caught on, and it makes me less productive on Macs and Windows. Macs are in some ways the worst, because it's Unix but it's not X11: almost, but not quite. (And to add to the pain, Mac keyboard shortcuts are nothing like xterm ones.)

Of course, if you do need to type some part of the cut-and-paste script learning and using keyboard shortcuts for that, particularly paste, are the way to go. Consider this simple script:

for host in $(< hostlist ) ; do
        # need to set EDITOR enviornment variable after sudo
	ssh -t $host "sudo env EDITOR=ex crontab -e"
done
# paste to ex:
#	:g /day$/ s, \* /, 1-5 /, | x 

One can copy that global command, which finds lines ending in "day" and edits the * before the / to be 1-5. For a crontab like:

MAILTO=cron-list@example.com
3 9 * * * /usr/local/bin/start-day
10 2 * * * /usr/local/bin/run-backup

This will change run every day at 9:03 am start-day program to only run on weekdays: 3 9 * * 1-5 /usr/local/bin/start-day and save the file. The for loop will log in to each host in the list of hosts, possibly prompting for an ssh password (depending on how you have that set up) and probably prompting for a sudo password (depending on how you have that set up). It would be quite reasonable to run the loop, enter passwords by hand, then hit <shift-insert> to use the default xterm paste primary keybinding to paste the script whereupon cron is updated, you are logged out, and move on to next host, so you end up just typing password<enter><shift-insert> password<enter><shift-insert> password<enter><shift-insert> ....

Some tricks of the trade that I've picked up over the years of doing this:

• Many things that don't immediately look cut-and-pastable can be made so by rethinking them. File edits with ex or ed are easier to script than vi; xdotool can be used to control things in windows.
• Whole lines are easier to copy than fragments, triple-click will select a line.
• Learn exactly how to include or not include trailing new lines in your copies, it can make a lot of difference.
• Use variables like you would in a regular script loop for things that will vary from run to run, even if each run is on a separate host. This lets you separate variable setting pastes from command run pastes.
• Setting variables with shell commands instead of manually makes for a smoother experience. grep, head, and tail, are good for selecting lines from output. cut, sed, and grep -o are good for selecting part of a line from output:

     host=$(hostname -f)
     homeuse=$(df /home | tail -1 | grep -o '[0-9]*%' | cut -f 1 -d %) 

• Some shell line break methods are more fragile than others. Consider:

  # Pasted as a group, both of these run, even if you only want the
  # second to run if the first succeeds.
  /bin/false
  /bin/true
  
  # Pasted as a group, the second will run only if the first succeeds
  /bin/true &&
  /bin/false
  
  # The backslash will escape the following whitespace character.
  # If you have a trailing space, that's the whitespace that will
  # be escaped, not the newline.
  [ -f "$somefile" ] && 
  	head "$somefile" \ 
  	| next command
  
  # Written like this, trailing whitespace is ignored, the pipe
  # alone is enough to keep the shell looking for more to do.
  [ -f "$somefile" ] && 
  	head "$somefile" | 
  	next command
  

And remember, cut-and-paste scripts are a good way to write a real script, slowly dealing with all the error cases to handle and best ways to make the code universal. Use them to compose your startup scripts, your cron jobs, your configuration management tooling, your runbooks, and coding your self-healing systems.