Bot Traffic, Again

One of annoying things I had happen last time this blog was in active use was getting hammered by a rogue bot. It has happened again.

Hits by bot:

Hits by non-bots: 38 unique User-Agents (across ~500 hits)

One user agent really stands out. And one other is suspicious. I'm talking about the two that hit my site more than world-famous Google.

I don't know everything MJ12bot does, but I do know one thing it does is power paid access to "incoming" links reports via "Majestic Site Explorer": "Access raw exports from £79.99 a month". So let me get this, you crawl sites to sell people lists of who links to them? Why should I waste my bandwidth giving you pages?

But clearly it is Megaindex that is abusive. At the .com version of the site I read "MegaIndex is a powerful and versatile competitive intelligence suite for online marketing, from SEO and PPC to social media and advertising research." Again, this is a bullshit use of my resources (bandwidth, web server CPU) for some commercial enterprise that cannot benefit me.

So: another new plugin is born, browser_block. Goodbye Megaindex. Goodbye Majestic.

New Plugins!

I've been thinking about writing a tag plug-in, and making that combined pagination / permalinks plug-in I mentioned earlier. But I put that off in order to write some smaller plugins to get myself more familiar with the code.

Both of the new plugins are very similar to the sample one in the plugins developer documentation. One of them fixes how I think blosxom should work: keep the templates out of the blog data directory. (Blosxom does not do that, so that you can have different templates in different subdirectories if you want. I don't want.) The other replaces the template reader to substitute new templates, but only on April Fool's Day (April 1^st).

plugins/flavour_dir takes a single configuration parameter to hold the location for the flavours. Now my git tree will match how the blog actually is. (I do not keep the blog posts in git.)

notused-plugins/april_fools takes a single configuration parameter to hold the name of the new flavour to use on April Fool's Day.

And while I was at it, I've updated the css file to make <code> and <pre> tags much more visible; fixed the styles to be more phone friendly (but there's more to do); added a copyright notice; removed all of the HTML validation errors, there are still some warnings.

Once I get tags and tag searching working, I think this will be much better. But it's coming along.

About Post Ordering

One thing that is tricky with Blosxom is that posts are (unless changed by plugin) ordered strictly by modification time. At first blush, this seems very reasonable. But eventually one finds problems.

A naïve backup and restore can totally thrash file modification times, and leave everything in a random unpredicatable order.

# don't backup Blosxom like this, restore will be broken
cp blog/* /mnt/backup-disk/blog/

Then there are subtler things. In my previous use of this blog, I replaced the sorting plugin to show most recent posts on first page, and alphabetal posts on all category pages. That suited my "bookmark blog" usage very well. It doesn't suit my current usage, and I disabled that sort when I restarted.

But when I restarted I also moved the hostname. And I edited the old posts that had references to the hostname. I edited the old posts.... I lost the original time stamps on those two posts, and I don't know when they are from. I probably have the timestamps in a tar backup somewhere, but that's not handy since it's old. (Panix has backups, too, but not in a format easy to get an old timesstamp from.) I "solved" that by using touch to change the dates back the same same month of my last posts. It's not right, but it's a lot more right than February 2020.

Enter my quick fix for now, a tool to save time stamps in a way that is very easy to restore from. I call it SAVE-DATES and it's a wrapper around a tool I have not used for anything else in many years: super ls (sls). Quoting the README I wrote for it:

In 1989 this program was released to the world in the form of being posted to the Usenet group comp.sources.unix. It was in volume 18 of the archives, which probably exist out on some dusty ftp server still. (But a quick search did not find such an ftp server in October 2017. The links run cold at the Internet Systems Consortium's sources.isc.org.)

Many of the features of sls were new at the time, but are less radical now. And the growth of interpreted languages for system admin use. Languages like Perl (described as a "replacement" for awk and sed, when posted to c.s.u, and archived in volume 13) and Python provide native access to stat(2). And now there's a straight-out stat(1) in Gnu coreutils.

I found this program in the archives somewhere around 1993 and immediately took a liking to it, and patched it many times for my own needs. Today I barely use it, sls having fallen to the wayside due to several factors: non-standard interface (the printf style output control isn't really printf(3)), the rampant buffer overrun problems in the code, more powerful all-in-one output features available with Perl.

The heart of sls that I still really like is the easy way I can turn a simple directory listing into usable code. And that's how my quick fix works. SAVE-DATES uses sls to create a shell script of touch commands that will restore dates. The whole thing can be run, or lines greped out to selectively (and more speedily) be used.

#!/bin/sh
#              [[CC]YY]MMDDhhmm[.ss]
export SLS_DATEFMT=%Y%m%d%H%M.%S
datadir=$PHTML/qz/data
savefile=$HOME/notes/blog/FIX-DATES
# use sls to make shell commands to reset date/time on files
slsoutstyle='touch -t %m %n'
find "$datadir" -type f -exec sls -p "$slsoutstyle" {} + > "$savefile"

The environment variable makes the date format the same as the date in format touch wants. Then I have the output format of sls include the fixed strings needed for touch to run, plus the modification time (%m) and name (%n). Note that the names are raw, so it relies on me picking "safe" names for post files, but that's my habit anyway. The output looks like this:

touch -t 202002222054.00 /net/u/3/e/eli/public_html/qz/data/blog/feb-2020-a-reboot.txt
touch -t 202002230325.45 /net/u/3/e/eli/public_html/qz/data/blog/how-its-built.txt
touch -t 202002240050.59 /net/u/3/e/eli/public_html/qz/data/blog/haircuts.txt
touch -t 202002250217.49 /net/u/3/e/eli/public_html/qz/data/blog/first-patch.txt
touch -t 202002270203.01 /net/u/3/e/eli/public_html/qz/data/blog/very-impressive-phone.txt

Date restore is easy-peasy now.

First patch!

A couple of things I want for here: permalinks to posts and pagination to browse through older posts. I had the cooluri plugin for permalinks and the pagination plugin for, well, pagination. Each worked on their own, but installed with all the others I was having issues.

I dug in and figured it out. I've renamed cooluri now to nowcooluri, this forces it after menu (otherwise on permalink pages the side menu counts are all broken), and have it set a global $use_permalink when it's filtering is in place. And if $use_permalink is set, disable the pagination in paginationqz (renamed just to show it's a local version).

And when $use_permalink is set, disable the category filtering in blosxom itself. See 2020-02-24-permalinks.patch

Longer term, I think I will combine the permalink and pagination into a single plug-in, so that on a page for a single entry I have have next/previous links to newer and older single entries.

PS: github.com/Eli-the-Bearded/qaz-blog/