Traditionally, one would make all the copies writeable and rely on either conventions of use or tree-diffing to periodically reintegrate the changes. Good tools for comparing filesystem trees are hard to come by, though, especially when your definition of “good” includes both readability and content analysis; identifying moves and copies would be nice too.
An improvement on this is to use a BTRFS snapshot as a common starting point, and then instantiate the two sides from there. Via the excellent btrbk, one can then keep reciprocal incremental snapshots of the two instances that both share the initial data and independently record changes on both sides. This workflow doesn’t remove the reintegration problem though.
Instead, I’ve recently simplified by having a writeable subvolume on machine A (the master copy) and subvolume-mounting a snapshot thereof on machine B (the read-only copy). This reduces complexity by giving me only one snapshot lineage to manage, but it does mean periodically remounting to switch machine B to use a more up-to-date snapshot.
This is where btram comes in: when run, it will spot subvolume mounts of outdated snapshots (i.e. where later-dated snapshots of the same set are available) and remount automatically. You can also dry-run to see what would happen without doing any actual remounting.
Since developing RotKraken, I’ve found it to have some fairly life-changing use cases, not least that it makes the job of organising my filesystem ridiculously fast and easy; suspected duplicate trees can be compared instantly with the included rkdiff tool before removal.
In this case, though, RK’s standard functionality was a lifesaver for checking both layout and content of rdfind’s dedupe results, except for one thing: there was a subtle library bug in the way.
Some time after running rk -i on the initial snapshot containing the raw filesystem copy, I discovered that some of the files had never initialised and, in fact, wouldn’t do so. They were all Icon^M files, which macOS software seems to use frequently.
After 60-90 minutes of investigation, I traced the problem to the Digest::MD5::File Perl library; it barfs on filenames containing trailing carriage returns, returning undef from its ->file_md5_hex method. The bug quickly turned out actually to be in the Digest::MD5->addfile API to which it delegates the hashing under the hood.
This hadn’t shown up in RK’s (fairly comprehensive) tests because, although they deliberately use pathological filenames containing all kinds of unusual characters, the names don’t end with a CR, which is where the problem actually shows up.
I worked around the problem by replacing the Digest::MD5::File->file_md5_hex call with an explicit open/slurp/close pattern that uses an instance of Digest::MD5 and calls ->md5_hex at the end. I verified correctness via the (now amended to use filenames with trailing CRs) tests.
After all this, RK worked perfectly on the dataset, allowing me (via the bundled rkdiff-stdin tool) to verify no difference between the copied files and the initial hash log I took from the source disk with md5deep. As a side benefit, RotKraken is now more practical to use on Mac system backups.
They say that “engineering works because it assumes it doesn’t”, thus the existence of test-driven development, which I used from the start for RotKraken. Despite also designing it never to touch file content, I wasn’t going to assume anything, so here’s how I carefully applied it to my data.
Note: You’ll see a couple of instances of the sync command below. This is because, at least on Ubuntu 20.04, BTRFS has minor bugs (presumably to do with making generation-number updates visible before they’re committed to disk) that sometimes make it act as if nothing on a subvolume has changed. While this doesn’t necessarily mean that snapshotting will use the out-of-date state, I prophylactically sync before snapshotting subvolumes I’ve just changed to be sure.
We first snapshot the data to provide a safety copy should anything bad happen:
sync
btrfs subvolume snapshot -r $PATH $SNAP_PRE
We run RotKraken on the data to do the initial hashing run:
rk -i $PATH
To guard against rsync mishaps, we take another read-only snapshot:
sync
btrfs subvolume snapshot -r $PATH $SNAP_POST
Finally, we use rsync’s handy tree-diffing mode to check that the only changes were to extended attributes:
rsync --delete -aHAXni $SNAP_PRE/ $SNAP_POST/ | grep -v '^.f........x'
Et voilà! We can now dispose of the snapshots (if we wish) and get on with more interesting things — with the assurance that any future change to the content of these files will be detectable.
]]>Christmas is traditionally a time for family and friends, but it’s also a convenient lull in the year’s pace, and I’ve developed a habit of using the period to get around to some of the more mundane IT-infrastructure tasks that are rarely urgent enough to do during the rest of the year, especially data management. This last Christmas, therefore, I took the step of trying BTRFS for the first time.
In retrospect, I made two mistakes. For one thing, the BTRFS version shipped with Ubuntu 16 was in the last stage of relative immaturity and I really should have upgraded to Ubuntu 20 before trying it. The far greater mistake, though, was to use its spanning feature to join two 8 TB SMR hard drives into a single 16 TB volume. In retrospect, this was a Very Bad Idea™ because some of the things you need to be able to do in a spanning/RAID setup — e.g. btrfs balance — cannot be rate-controlled. Not only do SMR drives tend to slow down after a few hundred gigabytes of continuous writes, some of them eventually grind to a halt, potentially leaving you with no way to get the filesystem into a consistent state.
These mistakes led to a nightmarish episode in which I got multiple levels down a stack of yaks, convinced myself that I had at least one faulty drive, and ended up with a broken filesystem that I didn’t have enough free space to run btrfs restore on. In the end I solved the last by splurging on server-grade WD Gold drives and managed to emerge from the chaos traumatised but without any actual data loss.
I threw BTRFS firmly in the sea, got everything shipshape again, and got on with my life.
Recently, the trauma faded enough for me to learn the correct lesson from all this.
When it comes to data management, there’s a whole category of concerns that seem both diverse and complex — until you realise that they all have the same answer.
Q: What filesystem should I use to minimise the chances of data loss?
Q: Should I get a UPS to guard against filesystem corruption during blackouts?
Q: Should I buy server-grade drives and/or make sure they’re CMR rather than SMR to preserve my data?
To illustrate:
“I want to use the most trustworthy filesystem possible.”
No you don’t; you want to use one that isn’t obviously hazardous, and think of it as disposable because you have enough redundancy.
“I want to make sure my power supply is as reliable as possible to minimise uncontrolled shutdowns.”
No you don’t; you want to evaluate the danger of acute data loss versus the real-world frequency of power-loss events, and assess the cost-to-benefit ratio of buying a UPS based on that.
“I want to use the most trustworthy drives money can buy.”
No you don’t; you want to make that decision based on rational considerations like (a) the amortised time cost of replacing faulty drives, (b) the financial cost, over time, of replacing cheap drives that might not last as long versus that cost for drives with a decent warranty.
“I really must get around to organising my decade-plus of accumulated digital junk so I can find things again and free up some space, and I need to do that first.”
NO, NO, NO!! (whacks speaker with rolled-up newspaper) What you really must do is urgently replicate that digital slagheap and achieve redundancy. Once that’s in place, THEN you can sift through it all knowing that the inevitable missteps won’t lose you anything.
I should make clear that the four above points are not strawmen but actually factor into my thinking. The important thing is that by removing the data-loss penalty for mistaken decisions, I don’t have to agonise so much and can take a “live and learn” approach.
While the irony of repeatedly banging on about redundancy isn’t lost on me, it bears repeating because it’s such an important paradigm shift. As for how much is enough, my current policy is 3 copies. Of course, this can be varied depending on the importance of the data, but I generally only do that for offlining data to tape. For online storage, I find it easier to apply a uniform policy, not least because it avoids incessant knapsacking difficulties.
Learning the correct lesson has meant overcoming my fear of BTRFS and other “experimental” filesystems, so I’m trying it again. This time it’s on Ubuntu 22, and definitely without any spanning or RAID that would get in the way of recovering from a single-drive failure. Then again, who cares? I have enough redundancy, so if at some future point spanning offers any value, I can reconsider it.
]]>A search yielded the more expensive but connected BX950UI, with full support on Windows, GNU/Linux, and macOS via the excellent APC UPS Daemon, so I went for it.
On its arrival, I charged it and spent a couple of hours doing physical hookup, which involved replacing the plugs on a couple of multiway power strips with IEC C13s. This article and the apcupsd manpage helped me get the basics going in another couple of hours, and I ran a battery calibration.
My theory is that this step doesn’t directly use power-consumption information, but records the drain curve of the battery with respect to total energy delivered to loads, allowing the UPS to correctly estimate remaining runtime for any load. I suspect that it’s worth recalibrating every 3 months or so, both to keep this profile updated and to give the battery itself enough exercise to avoid degrading too fast; I gather lead-acid batteries don’t like being left charged for too many months on end.
At first I began editing the /etc/apcupsd/apccontrol script directly, but this file needs to be replaceable on package upgrade, so the correct approach is to edit the scripts specific to each event, e.g. /etc/apcupsd/doshutdown. The next gotcha was that, not having read the above article properly, I missed the important step of setting ISCONFIGURED=yes in /etc/default/apcupsd, without which apcaccess would work — but not the daemon itself. I was alerted to this problem by the smoking gun that neither the /var/log/apcupsd.events nor /etc/apcupsd/powerfail files were being created during testing.
As well as the Ubuntu server (an Intel NUC), I have a Windows 10 desktop attached to the UPS. Since the latter sucks enough juice to reduce battery runtime to about ten minutes, it was crucial to arrange for it to shut down quickly on power loss in order to maximise the server’s uptime. To this end I put the following command in the /etc/apcupsd/onbattery script, just before the exit 0 line (to conceal any failure owing to e.g. the machine already being off), and using its IP address to dodge any transient name-resolution problems:
net rpc shutdown -t 45 -f -C "UPS shutdown" -I $IP_ADDRESS -U$USERNAME%$PASSWORD
This gives 45 seconds’ warning, just enough to run shutdown /a if there’s a pressing need to use the machine. Testing revealed a few gotchas: the first was that the remote-shutdown feature apparently (see later) requires the Remote Registry service to be running, which got me from one error message to another.
The second problem was the need for UAC auto-elevation on RPC connections; I fixed that by using regedit to create the DWORD value LocalAccountTokenFilterPolicy=1 inside the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System key.
I later discovered that the Remote Registry service was set to Automatic but not actually started — and thus unnecessary. I left it on Automatic anyway and started it, because there’s a nasty impasse one can trip over during cloning/migration of Windows builds where logon is blocked owing to a faulty MountedDevices configuration.
Another detail worth noting is that I don’t see the point of blocking remote logins on a GNU/Linux system that’s in the process of shutting down; it seems like a nannying feature to save admins from having their session nuked by the shutdown (not much of a benefit) and it takes away options in a time-critical situation. I therefore set this line in /etc/apcupsd/apcupsd.conf:
NOLOGON disable
Here’s how it looks, along with my comments.
Since I run a Git repository in the Ubuntu server’s /etc folder, the following gets rid of diff noise:
apctest.output
powerfail
Since I don’t check root’s mailbox on the server, I changed the export SYSADMIN line to refer to my external e-mail address.
Here I altered the following:
UPSNAME $NAME
UPSCABLE usb
UPSTYPE usb
DEVICE
POLLTIME 15
ONBATTERYDELAY 30
MINUTES 2
#BATTERYLEVEL 5
POLLTIME is the maximum time (in seconds) that apcupsd will take to notice a UPS event. I believe ONBATTERYDELAY is the time in seconds between one of apcupsd’s polls noticing power loss and the onbattery state being triggered; this is how I prevent very short outages from shutting down the Windows machine. MINUTES specifies how close to battery exhaustion the UPS can get before a shutdown is triggered; 2 minutes is generous for my server, which usually takes 10-20 seconds to shut down.
Using MINUTES made BATTERYLEVEL unnecessary, so I commented it out.
Here I added the remote-shutdown command for the Windows machine (see above).
I replicated the Windows remote-shutdown command here (again before the exit 0 line to conceal failure) so that, even if I choose to keep the machine on during power loss via shutdown /a, it will retry on battery exhaustion.
ISCONFIGURED=yes
One thing I must address is what I consider the biggest (only?) flaw in apcupsd: its failure to provide handling for communications failures, which might sabotage the whole setup by hiding a power loss. In theory, this could be worked around with a bunch of custom scripting, but I think the real solution would lie in apcupsd itself:
COMMFAILUREDELAY setting that allows comms to recover within some timespan without generating an event;The other major improvement to my setup would be to hibernate the machines in question instead of shutting them down. I looked into this for a while, but there are two problems:
net rpc shutdown command provides only shutdown and restart, not sleep or hibernate.I’ll probably deal with the former via an already-planned server rebuild, and the latter could be fixed by remotely invoking the native Windows SHUTDOWN command instead of using the Samba one, but since Windows 10 no longer has an inbuilt Telnet server this would require extra software and time.
Thanks to this modest amount of work, and some testing — sometimes using the TIMEOUT setting in /etc/apcupsd/apcupsd.conf to avoid running the battery all the way down — I no longer have to worry about power loss in the middle of an intense work session, fiddling around with manual Git-repository repairs, or touring the local construction sites in a blame-seeking rage. Yay!
First, let me backtrack. For some years now my approach to data integrity (i.e. combatting the surprisingly frequent bit rot) has revolved around md5deep. I’ve been storing logs from that tool with datasets so that I can check later — via another run and a diff — that no bits have flipped in the intervening time. The big problem with this approach has been that these logs quickly become out of date owing to file/directory moves and other day-to-day management, and this means that, even when diffing, there’s the problem of tying everything up across moves in order to check that the content hashes themselves haven’t changed.
“If only”, I thought to myself, “I could store the hash information WITH the files themselves! Then all the moving/renaming in the world won’t matter.” I just didn’t see how to do that.
While trying to research support for the DOS archive attribute via Samba on Linux™ (spoiler alert: there doesn’t seem to be any), I came across the concept of extended file attributes. “Holy mackerel!” I said to myself, “This is it!”
I dusted off my rusty Perl skills and bashed out a prototype with a friend in an hour or two. A week of work in two stints later, I had an initial stab at a tool with real value.
Apart from the benefits of attaching the metadata to the files — no more hash logs or verification difficulties from moves/renames — I’ve quickly found the tool to be a game-changer in other ways too. Firstly, just by separating the long-running initial hashing job from active decisionmaking, it’s proved to be a killer tool for manually deduplicating and organising my files. By using the export feature to generate hash logs for trees, I can use diff to instantly verify that suspected duplicate directories are actually identical and just delete one.
I look forward to doing more work over the next couple of months to polish this product and make it as useful as possible. Watch this space!
Addendum: The tool is named RotKraken.
]]>You might not be very aware of this. I’m a programmer and even I forget it most of the time, usually because a simple website I’m using is stuffed full of unnecessary JavaScript frameworks, or the desktop software I’m using is layered on too many heavyweight APIs. Deep, inefficient technology stacks are everywhere these days.
Much of the time, though, that isn’t the whole problem; it’s the fundamentals too. When I implemented two features for Drummer - my C++-based rhythm-programming software for Windows - their sheer blazing speed reminded me how much details matter.
One feature was rendering: the process of taking all the user’s rhythm programming and spitting out either a single mixed audio file or a parallel set of them (one per instrument) for inclusion in your DAW project. As an example, a rhythm track for a 5-minute song with 10 monophonic instruments involves generating (at a sample rate of 48 kHz) about half a gigabyte of data. Drummer can render that to a WAV file, or a set of them, in less than half a second on a modern machine with a typical cheap SATA SSD.
The second feature was the full-text query system for finding sound samples in your collection based on their metadata. It builds the indices from scratch for 4,500 sample files in 7 seconds on the same SSD, which is pretty neat, but searching is what really shines: a results list matching a keyword string takes about 100 microseconds to generate - that’s a tenth of a millisecond.
How did I achieve this? Through avoiding a few common habits:
This might all sound very C++-centric, but that’s not true. C# and many other languages can achieve nearly the same performance, and I’ve used these principles to make similarly hair-raising improvements to speed-critical web-service code in my day-to-day contract development. The biggest pitfall is the garbage collector: it frees you from explicit ownership concerns, at least for read-only data, but places some inconvenient constraints on memory layout and allocation strategy that can cause locality to suffer. If your buffers aren’t unnecessarily huge and you’re not allocating them on the fly, though, locality matters much less thanks to the CPU cache.
While C# now has unsafe and pointers to mitigate these problems, there’s a good argument that using them can badly compromise your code’s developer-compatibility, always a concern in attrition-prone organisations. The good news is that there’s usually no need to go this low-level unless you’re squeezing the last few percent on performance. With careful design, even vanilla “safe” C# can be much faster than you might think.