Recover from a pacman lock left behind by a power cut

A machine switched off mid-update leaves /var/lib/pacman/db.lck behind. Nothing
removed it, so every subsequent run deferred on it - one power cut would have
stopped updates permanently and silently, which on an unattended machine is the
worst outcome there is.

A lock older than the current boot is provably abandoned: no process that could
hold it still exists. Those are now removed and the interrupted upgrade is
repeated, with pacman reinstalling anything caught half-written. A lock that is
merely unheld within the same boot stays untouched and is only reported, since
removing it could corrupt a live transaction; the boot-time test is what makes
the difference between a proof and a guess. A fuser check is kept alongside it
so a backwards clock jump cannot make a live lock look abandoned.

Documented what each layer can actually promise: suspend and normal shutdown
are blocked by the existing inhibitor, a hard power-off cannot be prevented by
anything, and snap-pac's pre/post snapshots remain the backstop.

Makefile

@@ -8,7 +8,7 @@
 # Overridable so a packager can pass the version it is actually building
 # (`make VERSION=$pkgver`). The literal below is the fallback for builds
 # straight from a checkout, and is what a release tag has to carry.
-VERSION      ?= 1.0.5
+VERSION      ?= 1.0.8
 
 PREFIX       ?= /usr
 DESTDIR      ?=

README.md

@@ -55,8 +55,15 @@ business rebuilding somebody's system before being asked.
 | Flatpak | system and per-user installations |
 | AppImages | via [Gear Lever](https://github.com/mijorus/gearlever), if installed |
 
-`-git`/`-devel` AUR packages, cache trimming and orphan removal exist as
-options but are off by default.
+It also trims the pacman package cache after each run (`paccache`, keeping the
+3 most recent versions), because otherwise `/var/cache/pacman/pkg` grows
+forever — tens of gigabytes on a machine with a few large packages. Set
+`KeepOldPackages=1` if disk space matters more than the ability to downgrade.
+
+`-git`/`-devel` AUR packages and orphan removal exist as options but are off by
+default. Orphan removal deletes installed software, and "orphaned" only means
+nothing else depends on it — which is also true of something installed
+deliberately.
 
 ## When it holds back
 
@@ -116,6 +123,36 @@ A leftover `db.lck` from a crashed transaction is never deleted automatically
 guessing wrong there corrupts a live transaction. After it has been seen
 unheld on several consecutive runs, you get a notification instead.
 
+## What if the machine is switched off mid-update
+
+Three layers, in order of how much they can actually promise:
+
+**Suspend and a normal shutdown are blocked.** The run holds a
+`systemd-inhibit --what=sleep:shutdown --mode=block` lock, so closing the lid,
+picking "Shut down" from the menu or a short press of the power button will not
+interrupt a transaction — the desktop says something is still busy instead.
+
+**A hard power-off cannot be prevented by anything.** Holding the power button
+or pulling the plug cuts power in firmware. What limits the damage is that
+pacman's commit phase is short (about a minute even for a 200-package upgrade)
+and that most of a run is downloading, where an interruption costs nothing but
+a partial file.
+
+**The next run repairs it.** A `db.lck` left behind is detected and removed —
+but only when it is *provably* dead, meaning it is older than the current boot,
+so no process that could hold it still exists. The interrupted upgrade is then
+simply run again; pacman reinstalls anything that was caught half-written. A
+lock that is merely unheld within the same boot is never removed, only
+reported, because there the guess could be wrong.
+
+This last part matters more than it sounds: without it, a single power cut
+during an update would leave a lock file that makes every future run defer,
+and the machine would stop updating silently and permanently.
+
+On a Btrfs system with `snapper` and `snap-pac` — the CachyOS default — every
+pacman transaction is bracketed by a pre and post snapshot, so a genuinely
+broken upgrade can still be rolled back with `snapper rollback`.
+
 ## Configuration
 
 `/etc/cachy-auto-update/cachy-auto-update.conf`, one `Key=Value` per line, every

doc/cachy-auto-update.1.scd

@@ -100,6 +100,26 @@ running are skipped.
 The machine is never restarted automatically. When a kernel update makes a
 restart necessary, a notification says so.
 
+# INTERRUPTED UPDATES
+
+While a transaction is running, *cachy-auto-update* holds a
+*systemd-inhibit*(1) lock on _sleep_ and _shutdown_ in blocking mode, so a
+suspend, a lid close or a normal shutdown request cannot cut it short.
+
+A hard power-off - holding the power button, or losing mains power - is not
+preventable. On the next run a leftover _/var/lib/pacman/db.lck_ is removed if
+it is older than the current boot, since no process able to hold it can still
+exist; the upgrade is then repeated and pacman reinstalls whatever was caught
+half-written. A lock file that is unheld but was created during the current
+boot is reported rather than removed, because there is no way to prove it is
+abandoned.
+
+Without that recovery a single power cut would leave a lock that makes every
+subsequent run defer, silently stopping updates for good.
+
+On Btrfs with *snapper*(8) and *snap-pac*, each pacman transaction is bracketed
+by a pre and post snapshot, so a broken upgrade remains rollbackable.
+
 # FILES
 
 _/etc/cachy-auto-update/cachy-auto-update.conf_

po/cachy-auto-update.pot

@@ -254,3 +254,12 @@ msgstr ""
 #, c-format
 msgid "Held back: %s"
 msgstr ""
+
+msgid "The last run was stopped before it finished."
+msgstr ""
+
+msgid "Finishing an interrupted update"
+msgstr ""
+
+msgid "The last update was cut short, most likely because the machine was switched off. It is being finished now."
+msgstr ""

po/de.po

@@ -255,3 +255,12 @@ msgstr "%s konnte nicht aktualisiert werden und wurde übersprungen. Alles ander
 #, c-format
 msgid "Held back: %s"
 msgstr "Zurückgehalten: %s"
+
+msgid "The last run was stopped before it finished."
+msgstr "Der letzte Lauf wurde abgebrochen, bevor er fertig war."
+
+msgid "Finishing an interrupted update"
+msgstr "Abgebrochenes Update wird beendet"
+
+msgid "The last update was cut short, most likely because the machine was switched off. It is being finished now."
+msgstr "Das letzte Update wurde unterbrochen, vermutlich weil der Rechner ausgeschaltet wurde. Es wird jetzt zu Ende geführt."

res/config/cachy-auto-update.conf

@@ -66,16 +66,29 @@ AutoResolveConflicts=yes
 IgnorePkg=
 
 # ---------------------------------------------------------------------------
-# Housekeeping - all off by default
+# Housekeeping
 # ---------------------------------------------------------------------------
 
-# Trim the pacman package cache after an update.
-CleanCache=no
+# Trim the pacman package cache after an update: drop the older versions of
+# installed packages, and every cached version of packages that are no longer
+# installed. Nothing that is currently installed is ever touched, so this only
+# costs the ability to downgrade further back than KeepOldPackages.
+# Without it /var/cache/pacman/pkg grows forever - tens of gigabytes on a
+# machine with a few large packages.
+CleanCache=yes
 
-# How many old versions per package to keep when CleanCache is on.
+# How many old versions per package to keep when CleanCache is on. 3 is the
+# Arch default and leaves room to downgrade. Set it to 1 if disk space matters
+# more than that; on a machine with a handful of multi-gigabyte packages the
+# difference is easily ten gigabytes.
 KeepOldPackages=3
 
-# Remove packages that were installed as dependencies and are no longer needed.
+# Remove packages nothing depends on any more: pacman packages that were pulled
+# in as dependencies and are now unreferenced, plus Flatpak runtimes no
+# installed application uses.
+# Off by default, and unlike CleanCache this one deletes installed software:
+# "orphaned" only means no other package requires it, which is also true of
+# something installed deliberately as a dependency of nothing.
 RemoveOrphans=no
 
 # ---------------------------------------------------------------------------

src/cachy-auto-update-run

@@ -63,6 +63,21 @@ fi
 cau_config_load
 cau_log_open
 
+# Record an interruption rather than leaving the previous run's verdict behind.
+# Without this, killing an interactive run leaves last_result at whatever it was
+# before - so the menu can keep reporting a problem from hours ago while the
+# machine is in fact fully up to date, which is worse than saying nothing.
+# pacman makes the commit phase itself uninterruptible, so the packages either
+# all landed or none did; only our own bookkeeping is at risk here.
+_cau_interrupted() {
+	cau_error "Update run interrupted ($1)"
+	cau_state_write last_result interrupted
+	exit 130
+}
+trap '_cau_interrupted SIGINT'  INT
+trap '_cau_interrupted SIGTERM' TERM
+trap '_cau_interrupted SIGHUP'  HUP
+
 # ---------------------------------------------------------------------------
 # Should this run happen at all?
 # ---------------------------------------------------------------------------
@@ -102,6 +117,15 @@ if (( ! FORCE )); then
 	fi
 fi
 
+# A lock left behind by a power cut during a previous update. It is cleared
+# before the busy check, because otherwise every future run would defer on it
+# forever and the machine would quietly stop updating.
+if cau_recover_stale_lock; then
+	cau_notify normal \
+		"Finishing an interrupted update" \
+		"The last update was cut short, most likely because the machine was switched off. It is being finished now."
+fi
+
 # This one is checked even with --force: proceeding anyway would just hand the
 # user a lock error instead of doing anything useful.
 CAU_SKIP_REASON=''

src/lib/common.sh

@@ -91,7 +91,14 @@ cau_msg_in() {
 # Output and logging
 # ---------------------------------------------------------------------------
 
-if [[ -t 1 && -z ${NO_COLOR:-} ]]; then
+# Is a person watching? Decided once, here, while stdout is still whatever the
+# process was started with. Testing `-t 1` at the point of use is unreliable:
+# any function called through $(...) or <(...) sees a pipe on stdout and would
+# conclude nobody is there.
+CAU_INTERACTIVE=''
+[[ -t 1 ]] && CAU_INTERACTIVE=1
+
+if [[ -n $CAU_INTERACTIVE && -z ${NO_COLOR:-} ]]; then
 	CAU_C_RESET=$'\033[0m'
 	CAU_C_BOLD=$'\033[1m'
 	CAU_C_DIM=$'\033[2m'
@@ -132,10 +139,19 @@ cau_log_open() {
 	CAU_LOG_OPEN=1
 }
 
-# Runs a command, streaming its combined output into the run log. Returns the
+# Runs a command, capturing its combined output in the run log. Returns the
 # command's exit status.
+#
+# When a person is watching - `cachy-auto-update run` from a terminal - the
+# output is shown as well. Building an AUR package or pulling a few hundred
+# megabytes of Flatpak can take minutes, and silence for that long is
+# indistinguishable from a hang.
 cau_run_logged() {
 	if [[ -n ${CAU_LOG_OPEN:-} ]]; then
+		if [[ -n $CAU_INTERACTIVE ]]; then
+			"$@" 2>&1 | tee -a "$CAU_RUNLOG"
+			return "${PIPESTATUS[0]}"
+		fi
 		"$@" >> "$CAU_RUNLOG" 2>&1
 	else
 		"$@" >&2

src/lib/config.sh

@@ -96,7 +96,7 @@ cau_config_load() {
 	CFG_APPIMAGE=no;              cau_config_bool UpdateAppImages      yes && CFG_APPIMAGE=yes
 	CFG_DEVEL=no;                 cau_config_bool UpdateDevel          no  && CFG_DEVEL=yes
 	CFG_RESOLVE_CONFLICTS=no;     cau_config_bool AutoResolveConflicts yes && CFG_RESOLVE_CONFLICTS=yes
-	CFG_CLEAN_CACHE=no;           cau_config_bool CleanCache           no  && CFG_CLEAN_CACHE=yes
+	CFG_CLEAN_CACHE=no;           cau_config_bool CleanCache           yes && CFG_CLEAN_CACHE=yes
 	CFG_REMOVE_ORPHANS=no;        cau_config_bool RemoveOrphans        no  && CFG_REMOVE_ORPHANS=yes
 	CFG_NOTIFY_SUCCESS=no;        cau_config_bool NotifyOnSuccess      yes && CFG_NOTIFY_SUCCESS=yes
 	CFG_NOTIFY_ERROR=no;          cau_config_bool NotifyOnError        yes && CFG_NOTIFY_ERROR=yes

src/lib/locks.sh

@@ -60,11 +60,55 @@ cau_package_manager_busy() {
 	return 1
 }
 
+# cau_pacman_lock_is_stale
+# True only when the lock provably cannot belong to anything alive.
+#
+# The rigorous test is the boot time: no process that existed before the
+# current boot can still be running, so a db.lck older than boot is abandoned
+# by definition - which is exactly what a power cut during an update leaves
+# behind. A lock that is merely unheld *within* this boot is not provable in
+# the same way, so it is only reported (see cau_track_stale_lock) and never
+# removed; guessing wrong there would corrupt a live transaction.
+#
+# The fuser check is kept as a second condition purely to survive a backwards
+# clock jump making a live lock look pre-boot.
+cau_pacman_lock_is_stale() {
+	local boot lock
+
+	[[ -e $CAU_PACMAN_LOCK ]] || return 1
+
+	boot="$(awk '/^btime /{print $2}' /proc/stat 2>/dev/null)"
+	[[ $boot =~ ^[0-9]+$ ]] || return 1
+
+	lock="$(stat -c %Y "$CAU_PACMAN_LOCK" 2>/dev/null)" || return 1
+	[[ $lock =~ ^[0-9]+$ ]] || return 1
+
+	(( lock < boot )) || return 1
+	[[ -z "$(cau_pacman_lock_holder)" ]]
+}
+
+# cau_recover_stale_lock
+# Clears a provably abandoned lock so an interrupted update can be finished on
+# the next run. Without this, one power cut during an update stops every future
+# update permanently and silently - the worst possible outcome for a machine
+# nobody is watching.
+cau_recover_stale_lock() {
+	cau_pacman_lock_is_stale || return 1
+
+	cau_warn "Found a pacman lock older than this boot - an update was cut short"
+	rm -f "$CAU_PACMAN_LOCK" 2>/dev/null || {
+		cau_error "Could not remove the stale pacman lock"
+		return 1
+	}
+	cau_state_clear stale_lock_count
+	cau_info "Stale lock removed; the interrupted update will be finished now"
+	return 0
+}
+
 # cau_track_stale_lock
-# A db.lck with no process behind it is left over from a crashed transaction.
-# Removing it automatically would be reckless - if the guess is wrong it
-# corrupts a live transaction - so instead it is counted, and after enough
-# consecutive sightings the user is told to clean it up.
+# A db.lck with no process behind it but created during this boot: a crashed
+# pacman rather than a power cut. Not provable, so it is counted, and after
+# enough consecutive sightings the user is told to clean it up.
 CAU_STALE_LOCK_RUNS=3
 
 cau_track_stale_lock() {

src/lib/menu.sh

@@ -81,6 +81,10 @@ cau_ui_status() {
 		printf '\n  %s%s%s\n' "$CAU_C_YELLOW" \
 			"$(cau_msg "The last run reported a problem - see 'cachy-auto-update log'.")" \
 			"$CAU_C_RESET"
+	elif [[ $result == interrupted ]]; then
+		printf '\n  %s%s%s\n' "$CAU_C_YELLOW" \
+			"$(cau_msg "The last run was stopped before it finished.")" \
+			"$CAU_C_RESET"
 	fi
 	if [[ $reboot == 1 ]]; then
 		printf '\n  %s%s%s\n' "$CAU_C_YELLOW" \

src/lib/pkg_flatpak.sh

@@ -63,7 +63,10 @@ cau_flatpak_update() {
 		fi
 	done < <(cau_human_users)
 
-	if [[ $CFG_CLEAN_CACHE == yes ]]; then
+	# Unused runtimes are the Flatpak equivalent of orphaned packages - this is
+	# removal of installed software, not cache trimming, so it belongs behind
+	# RemoveOrphans rather than CleanCache.
+	if [[ $CFG_REMOVE_ORPHANS == yes ]]; then
 		cau_run_logged flatpak uninstall --system --unused --noninteractive --assumeyes || true
 	fi
 

src/lib/pkg_pacman.sh

@@ -15,13 +15,35 @@ CAU_PACMAN_HELD=''
 
 # Base flags for every unattended pacman invocation.
 cau_pacman_flags() {
-	printf '%s\n' --noconfirm --color never --noprogressbar --disable-download-timeout
+	printf '%s\n' --noconfirm --color never --disable-download-timeout
+
+	# A progress bar is worth having when somebody is watching a `run` from a
+	# terminal; in the timer's log it is only carriage-return noise.
+	[[ -n $CAU_INTERACTIVE ]] || printf '%s\n' --noprogressbar
+
 	local pkg
 	for pkg in $CFG_IGNORE_PKG; do
 		printf '%s\n' --ignore "$pkg"
 	done
 }
 
+# _cau_pacman_exec <logfile> <pacman args...>
+# Captures pacman's output for classification, and streams it as well when a
+# person is watching. Upgrading a few hundred packages takes minutes; without
+# this an interactive run shows one line and then nothing at all, which is
+# indistinguishable from a hang and invites someone to kill it mid-transaction.
+_cau_pacman_exec() {
+	local log="$1"
+	shift
+
+	if [[ -n $CAU_INTERACTIVE ]]; then
+		pacman "$@" 2>&1 | tee "$log"
+		return "${PIPESTATUS[0]}"
+	fi
+
+	pacman "$@" > "$log" 2>&1
+}
+
 # cau_pacman_pending
 # Fills CAU_PACMAN_PENDING and returns 1 when there is nothing to do.
 cau_pacman_pending() {
@@ -121,7 +143,7 @@ cau_pacman_update() {
 	local attempt=0 b
 
 	while true; do
-		if pacman -Syu "${flags[@]}" "${extra[@]}" > "$log" 2>&1; then
+		if _cau_pacman_exec "$log" -Syu "${flags[@]}" "${extra[@]}"; then
 			cat "$log" >> "$CAU_RUNLOG" 2>/dev/null
 			grep -E '^(removing|replacing) ' "$log" 2>/dev/null \
 				| while read -r line; do cau_info "  $line"; done
@@ -232,7 +254,14 @@ cau_pacman_cleanup() {
 	fi
 
 	if [[ $CFG_CLEAN_CACHE == yes ]] && cau_have paccache; then
-		cau_info "Trimming the package cache"
+		# paccache's dry run reports what a real run would reclaim; logging it
+		# first is the only way to tell afterwards whether trimming is doing
+		# anything, since the real run says little.
+		local reclaim
+		reclaim="$( { paccache -d --nocolor -k"$CFG_KEEP_OLD"; paccache -du --nocolor -k0; } 2>&1 \
+			| grep -oE 'disk space saved: [^)]*' | paste -sd', ' -)"
+
+		cau_info "Trimming the package cache (keeping $CFG_KEEP_OLD old version(s))${reclaim:+ - $reclaim}"
 		cau_run_logged paccache -r --nocolor -k"$CFG_KEEP_OLD"  || cau_warn "paccache -r failed"
 		cau_run_logged paccache -ru --nocolor -k0               || cau_warn "paccache -ru failed"
 	fi