Recover from a pacman lock left behind by a power cut

A machine switched off mid-update leaves /var/lib/pacman/db.lck behind. Nothing
removed it, so every subsequent run deferred on it - one power cut would have
stopped updates permanently and silently, which on an unattended machine is the
worst outcome there is.

A lock older than the current boot is provably abandoned: no process that could
hold it still exists. Those are now removed and the interrupted upgrade is
repeated, with pacman reinstalling anything caught half-written. A lock that is
merely unheld within the same boot stays untouched and is only reported, since
removing it could corrupt a live transaction; the boot-time test is what makes
the difference between a proof and a guess. A fuser check is kept alongside it
so a backwards clock jump cannot make a live lock look abandoned.

Documented what each layer can actually promise: suspend and normal shutdown
are blocked by the existing inhibitor, a hard power-off cannot be prevented by
anything, and snap-pac's pre/post snapshots remain the backstop.

Makefile

@@ -5,7 +5,10 @@
 # targets degrade to a no-op when msgfmt/scdoc are missing, so the tree stays
 # usable for development without the build dependencies installed.
 
-VERSION      := 1.0.0
+# Overridable so a packager can pass the version it is actually building
+# (`make VERSION=$pkgver`). The literal below is the fallback for builds
+# straight from a checkout, and is what a release tag has to carry.
+VERSION      ?= 1.0.8
 
 PREFIX       ?= /usr
 DESTDIR      ?=

README.md

@@ -55,8 +55,15 @@ business rebuilding somebody's system before being asked.
 | Flatpak | system and per-user installations |
 | AppImages | via [Gear Lever](https://github.com/mijorus/gearlever), if installed |
 
-`-git`/`-devel` AUR packages, cache trimming and orphan removal exist as
-options but are off by default.
+It also trims the pacman package cache after each run (`paccache`, keeping the
+3 most recent versions), because otherwise `/var/cache/pacman/pkg` grows
+forever — tens of gigabytes on a machine with a few large packages. Set
+`KeepOldPackages=1` if disk space matters more than the ability to downgrade.
+
+`-git`/`-devel` AUR packages and orphan removal exist as options but are off by
+default. Orphan removal deletes installed software, and "orphaned" only means
+nothing else depends on it — which is also true of something installed
+deliberately.
 
 ## When it holds back
 
@@ -116,6 +123,36 @@ A leftover `db.lck` from a crashed transaction is never deleted automatically
 guessing wrong there corrupts a live transaction. After it has been seen
 unheld on several consecutive runs, you get a notification instead.
 
+## What if the machine is switched off mid-update
+
+Three layers, in order of how much they can actually promise:
+
+**Suspend and a normal shutdown are blocked.** The run holds a
+`systemd-inhibit --what=sleep:shutdown --mode=block` lock, so closing the lid,
+picking "Shut down" from the menu or a short press of the power button will not
+interrupt a transaction — the desktop says something is still busy instead.
+
+**A hard power-off cannot be prevented by anything.** Holding the power button
+or pulling the plug cuts power in firmware. What limits the damage is that
+pacman's commit phase is short (about a minute even for a 200-package upgrade)
+and that most of a run is downloading, where an interruption costs nothing but
+a partial file.
+
+**The next run repairs it.** A `db.lck` left behind is detected and removed —
+but only when it is *provably* dead, meaning it is older than the current boot,
+so no process that could hold it still exists. The interrupted upgrade is then
+simply run again; pacman reinstalls anything that was caught half-written. A
+lock that is merely unheld within the same boot is never removed, only
+reported, because there the guess could be wrong.
+
+This last part matters more than it sounds: without it, a single power cut
+during an update would leave a lock file that makes every future run defer,
+and the machine would stop updating silently and permanently.
+
+On a Btrfs system with `snapper` and `snap-pac` — the CachyOS default — every
+pacman transaction is bracketed by a pre and post snapshot, so a genuinely
+broken upgrade can still be rolled back with `snapper rollback`.
+
 ## Configuration
 
 `/etc/cachy-auto-update/cachy-auto-update.conf`, one `Key=Value` per line, every

doc/cachy-auto-update.1.scd

@@ -100,6 +100,26 @@ running are skipped.
 The machine is never restarted automatically. When a kernel update makes a
 restart necessary, a notification says so.
 
+# INTERRUPTED UPDATES
+
+While a transaction is running, *cachy-auto-update* holds a
+*systemd-inhibit*(1) lock on _sleep_ and _shutdown_ in blocking mode, so a
+suspend, a lid close or a normal shutdown request cannot cut it short.
+
+A hard power-off - holding the power button, or losing mains power - is not
+preventable. On the next run a leftover _/var/lib/pacman/db.lck_ is removed if
+it is older than the current boot, since no process able to hold it can still
+exist; the upgrade is then repeated and pacman reinstalls whatever was caught
+half-written. A lock file that is unheld but was created during the current
+boot is reported rather than removed, because there is no way to prove it is
+abandoned.
+
+Without that recovery a single power cut would leave a lock that makes every
+subsequent run defer, silently stopping updates for good.
+
+On Btrfs with *snapper*(8) and *snap-pac*, each pacman transaction is bracketed
+by a pre and post snapshot, so a broken upgrade remains rollbackable.
+
 # FILES
 
 _/etc/cachy-auto-update/cachy-auto-update.conf_

po/cachy-auto-update.pot

@@ -134,7 +134,7 @@ msgstr ""
 msgid "Quit"
 msgstr ""
 
-msgid "Press Enter to continue..."
+msgid "Press any key to continue..."
 msgstr ""
 
 #. Commands
@@ -243,3 +243,23 @@ msgstr ""
 
 msgid "pacman's lock file looks left over from an interrupted update. Updates are paused until it is cleared."
 msgstr ""
+
+msgid "Some packages were held back"
+msgstr ""
+
+#, c-format
+msgid "%s could not be updated and was skipped. Everything else is up to date."
+msgstr ""
+
+#, c-format
+msgid "Held back: %s"
+msgstr ""
+
+msgid "The last run was stopped before it finished."
+msgstr ""
+
+msgid "Finishing an interrupted update"
+msgstr ""
+
+msgid "The last update was cut short, most likely because the machine was switched off. It is being finished now."
+msgstr ""

po/de.po

@@ -135,8 +135,8 @@ msgstr "Aktuelle Bedingungen anzeigen"
 msgid "Quit"
 msgstr "Beenden"
 
-msgid "Press Enter to continue..."
-msgstr "Zum Fortfahren Enter drücken …"
+msgid "Press any key to continue..."
+msgstr "Zum Fortfahren eine Taste drücken …"
 
 #. Commands
 msgid "Automatic updates are on."
@@ -244,3 +244,23 @@ msgstr "Paketdatenbank gesperrt"
 
 msgid "pacman's lock file looks left over from an interrupted update. Updates are paused until it is cleared."
 msgstr "Die Sperrdatei von pacman scheint von einem abgebrochenen Update übrig zu sein. Bis sie entfernt ist, pausieren die Updates."
+
+msgid "Some packages were held back"
+msgstr "Einige Pakete wurden zurückgehalten"
+
+#, c-format
+msgid "%s could not be updated and was skipped. Everything else is up to date."
+msgstr "%s konnte nicht aktualisiert werden und wurde übersprungen. Alles andere ist aktuell."
+
+#, c-format
+msgid "Held back: %s"
+msgstr "Zurückgehalten: %s"
+
+msgid "The last run was stopped before it finished."
+msgstr "Der letzte Lauf wurde abgebrochen, bevor er fertig war."
+
+msgid "Finishing an interrupted update"
+msgstr "Abgebrochenes Update wird beendet"
+
+msgid "The last update was cut short, most likely because the machine was switched off. It is being finished now."
+msgstr "Das letzte Update wurde unterbrochen, vermutlich weil der Rechner ausgeschaltet wurde. Es wird jetzt zu Ende geführt."

res/config/cachy-auto-update.conf

@@ -66,16 +66,29 @@ AutoResolveConflicts=yes
 IgnorePkg=
 
 # ---------------------------------------------------------------------------
-# Housekeeping - all off by default
+# Housekeeping
 # ---------------------------------------------------------------------------
 
-# Trim the pacman package cache after an update.
-CleanCache=no
+# Trim the pacman package cache after an update: drop the older versions of
+# installed packages, and every cached version of packages that are no longer
+# installed. Nothing that is currently installed is ever touched, so this only
+# costs the ability to downgrade further back than KeepOldPackages.
+# Without it /var/cache/pacman/pkg grows forever - tens of gigabytes on a
+# machine with a few large packages.
+CleanCache=yes
 
-# How many old versions per package to keep when CleanCache is on.
+# How many old versions per package to keep when CleanCache is on. 3 is the
+# Arch default and leaves room to downgrade. Set it to 1 if disk space matters
+# more than that; on a machine with a handful of multi-gigabyte packages the
+# difference is easily ten gigabytes.
 KeepOldPackages=3
 
-# Remove packages that were installed as dependencies and are no longer needed.
+# Remove packages nothing depends on any more: pacman packages that were pulled
+# in as dependencies and are now unreferenced, plus Flatpak runtimes no
+# installed application uses.
+# Off by default, and unlike CleanCache this one deletes installed software:
+# "orphaned" only means no other package requires it, which is also true of
+# something installed deliberately as a dependency of nothing.
 RemoveOrphans=no
 
 # ---------------------------------------------------------------------------

src/cachy-auto-update

@@ -170,7 +170,9 @@ cau_do_status() {
 
 cau_do_run() {
 	cau_need_root
-	exec "$CAU_LIBEXECDIR/cachy-auto-update-run" "$@"
+	# Not exec'd: this is also called from the menu, which has to survive the
+	# run and redraw afterwards.
+	"$CAU_LIBEXECDIR/cachy-auto-update-run" "$@"
 }
 
 cau_do_log() {

src/cachy-auto-update-run

@@ -40,6 +40,13 @@ if [[ $EUID -ne 0 ]]; then
 	exit 2
 fi
 
+# Force a neutral locale here rather than relying on the unit's Environment=,
+# so a run started by hand behaves exactly like one started by the timer.
+# pacman failures are classified by matching its output, and on a German system
+# that output is German. Text aimed at a person does not come through here - a
+# notification is rendered in the recipient's own locale by cau_msg_in.
+export LC_ALL=C LANGUAGE=
+
 # Re-exec under an inhibitor so a suspend or shutdown cannot land in the middle
 # of a pacman transaction and leave the database half-written. Done before
 # anything else so the whole pass is covered, including the lock.
@@ -56,6 +63,21 @@ fi
 cau_config_load
 cau_log_open
 
+# Record an interruption rather than leaving the previous run's verdict behind.
+# Without this, killing an interactive run leaves last_result at whatever it was
+# before - so the menu can keep reporting a problem from hours ago while the
+# machine is in fact fully up to date, which is worse than saying nothing.
+# pacman makes the commit phase itself uninterruptible, so the packages either
+# all landed or none did; only our own bookkeeping is at risk here.
+_cau_interrupted() {
+	cau_error "Update run interrupted ($1)"
+	cau_state_write last_result interrupted
+	exit 130
+}
+trap '_cau_interrupted SIGINT'  INT
+trap '_cau_interrupted SIGTERM' TERM
+trap '_cau_interrupted SIGHUP'  HUP
+
 # ---------------------------------------------------------------------------
 # Should this run happen at all?
 # ---------------------------------------------------------------------------
@@ -95,6 +117,15 @@ if (( ! FORCE )); then
 	fi
 fi
 
+# A lock left behind by a power cut during a previous update. It is cleared
+# before the busy check, because otherwise every future run would defer on it
+# forever and the machine would quietly stop updating.
+if cau_recover_stale_lock; then
+	cau_notify normal \
+		"Finishing an interrupted update" \
+		"The last update was cut short, most likely because the machine was switched off. It is being finished now."
+fi
+
 # This one is checked even with --force: proceeding anyway would just hand the
 # user a lock error instead of doing anything useful.
 CAU_SKIP_REASON=''
@@ -195,6 +226,22 @@ else
 	fi
 fi
 
+# Packages skipped so the rest of the upgrade could proceed. The notification
+# only fires when the set changes: a blocker waiting on an upstream fix would
+# otherwise produce the same message every single day.
+prev_held="$(cau_state_read held_back '')"
+if [[ -n $CAU_PACMAN_HELD ]]; then
+	cau_state_write held_back "$CAU_PACMAN_HELD"
+	cau_warn "Held back: $CAU_PACMAN_HELD"
+	if [[ $CAU_PACMAN_HELD != "$prev_held" && $CFG_NOTIFY_ERROR == yes ]]; then
+		cau_notify normal "Some packages were held back" \
+			"%s could not be updated and was skipped. Everything else is up to date." \
+			"$CAU_PACMAN_HELD"
+	fi
+else
+	cau_state_clear held_back
+fi
+
 if cau_pacman_reboot_needed; then
 	cau_state_write reboot_needed 1
 	cau_info "A kernel update needs a restart"

src/lib/common.sh

@@ -91,7 +91,14 @@ cau_msg_in() {
 # Output and logging
 # ---------------------------------------------------------------------------
 
-if [[ -t 1 && -z ${NO_COLOR:-} ]]; then
+# Is a person watching? Decided once, here, while stdout is still whatever the
+# process was started with. Testing `-t 1` at the point of use is unreliable:
+# any function called through $(...) or <(...) sees a pipe on stdout and would
+# conclude nobody is there.
+CAU_INTERACTIVE=''
+[[ -t 1 ]] && CAU_INTERACTIVE=1
+
+if [[ -n $CAU_INTERACTIVE && -z ${NO_COLOR:-} ]]; then
 	CAU_C_RESET=$'\033[0m'
 	CAU_C_BOLD=$'\033[1m'
 	CAU_C_DIM=$'\033[2m'
@@ -132,10 +139,19 @@ cau_log_open() {
 	CAU_LOG_OPEN=1
 }
 
-# Runs a command, streaming its combined output into the run log. Returns the
+# Runs a command, capturing its combined output in the run log. Returns the
 # command's exit status.
+#
+# When a person is watching - `cachy-auto-update run` from a terminal - the
+# output is shown as well. Building an AUR package or pulling a few hundred
+# megabytes of Flatpak can take minutes, and silence for that long is
+# indistinguishable from a hang.
 cau_run_logged() {
 	if [[ -n ${CAU_LOG_OPEN:-} ]]; then
+		if [[ -n $CAU_INTERACTIVE ]]; then
+			"$@" 2>&1 | tee -a "$CAU_RUNLOG"
+			return "${PIPESTATUS[0]}"
+		fi
 		"$@" >> "$CAU_RUNLOG" 2>&1
 	else
 		"$@" >&2
@@ -146,11 +162,18 @@ cau_run_logged() {
 # Terminal helpers for the CLI
 # ---------------------------------------------------------------------------
 
+# Set by cau_bad and cau_note. The menu redraws immediately after an action,
+# which would wipe the screen; this marks that something was printed that the
+# user still has to read, so only those cases wait for a keypress. A plain
+# success needs no acknowledgement - the status block at the top of the menu
+# already shows the new state.
+CAU_UI_NEEDS_ACK=''
+
 cau_say()  { printf '%s\n' "$*"; }
 cau_head() { printf '\n%s%s%s\n\n' "$CAU_C_BOLD$CAU_C_BLUE" "$*" "$CAU_C_RESET"; }
 cau_ok()   { printf '%s✔%s %s\n' "$CAU_C_GREEN" "$CAU_C_RESET" "$*"; }
-cau_bad()  { printf '%s✘%s %s\n' "$CAU_C_RED" "$CAU_C_RESET" "$*" >&2; }
-cau_note() { printf '%s•%s %s\n' "$CAU_C_DIM" "$CAU_C_RESET" "$*"; }
+cau_bad()  { CAU_UI_NEEDS_ACK=1; printf '%s✘%s %s\n' "$CAU_C_RED" "$CAU_C_RESET" "$*" >&2; }
+cau_note() { CAU_UI_NEEDS_ACK=1; printf '%s•%s %s\n' "$CAU_C_DIM" "$CAU_C_RESET" "$*"; }
 
 # ---------------------------------------------------------------------------
 # State files

src/lib/config.sh

@@ -96,7 +96,7 @@ cau_config_load() {
 	CFG_APPIMAGE=no;              cau_config_bool UpdateAppImages      yes && CFG_APPIMAGE=yes
 	CFG_DEVEL=no;                 cau_config_bool UpdateDevel          no  && CFG_DEVEL=yes
 	CFG_RESOLVE_CONFLICTS=no;     cau_config_bool AutoResolveConflicts yes && CFG_RESOLVE_CONFLICTS=yes
-	CFG_CLEAN_CACHE=no;           cau_config_bool CleanCache           no  && CFG_CLEAN_CACHE=yes
+	CFG_CLEAN_CACHE=no;           cau_config_bool CleanCache           yes && CFG_CLEAN_CACHE=yes
 	CFG_REMOVE_ORPHANS=no;        cau_config_bool RemoveOrphans        no  && CFG_REMOVE_ORPHANS=yes
 	CFG_NOTIFY_SUCCESS=no;        cau_config_bool NotifyOnSuccess      yes && CFG_NOTIFY_SUCCESS=yes
 	CFG_NOTIFY_ERROR=no;          cau_config_bool NotifyOnError        yes && CFG_NOTIFY_ERROR=yes

src/lib/locks.sh

@@ -60,11 +60,55 @@ cau_package_manager_busy() {
 	return 1
 }
 
+# cau_pacman_lock_is_stale
+# True only when the lock provably cannot belong to anything alive.
+#
+# The rigorous test is the boot time: no process that existed before the
+# current boot can still be running, so a db.lck older than boot is abandoned
+# by definition - which is exactly what a power cut during an update leaves
+# behind. A lock that is merely unheld *within* this boot is not provable in
+# the same way, so it is only reported (see cau_track_stale_lock) and never
+# removed; guessing wrong there would corrupt a live transaction.
+#
+# The fuser check is kept as a second condition purely to survive a backwards
+# clock jump making a live lock look pre-boot.
+cau_pacman_lock_is_stale() {
+	local boot lock
+
+	[[ -e $CAU_PACMAN_LOCK ]] || return 1
+
+	boot="$(awk '/^btime /{print $2}' /proc/stat 2>/dev/null)"
+	[[ $boot =~ ^[0-9]+$ ]] || return 1
+
+	lock="$(stat -c %Y "$CAU_PACMAN_LOCK" 2>/dev/null)" || return 1
+	[[ $lock =~ ^[0-9]+$ ]] || return 1
+
+	(( lock < boot )) || return 1
+	[[ -z "$(cau_pacman_lock_holder)" ]]
+}
+
+# cau_recover_stale_lock
+# Clears a provably abandoned lock so an interrupted update can be finished on
+# the next run. Without this, one power cut during an update stops every future
+# update permanently and silently - the worst possible outcome for a machine
+# nobody is watching.
+cau_recover_stale_lock() {
+	cau_pacman_lock_is_stale || return 1
+
+	cau_warn "Found a pacman lock older than this boot - an update was cut short"
+	rm -f "$CAU_PACMAN_LOCK" 2>/dev/null || {
+		cau_error "Could not remove the stale pacman lock"
+		return 1
+	}
+	cau_state_clear stale_lock_count
+	cau_info "Stale lock removed; the interrupted update will be finished now"
+	return 0
+}
+
 # cau_track_stale_lock
-# A db.lck with no process behind it is left over from a crashed transaction.
-# Removing it automatically would be reckless - if the guess is wrong it
-# corrupts a live transaction - so instead it is counted, and after enough
-# consecutive sightings the user is told to clean it up.
+# A db.lck with no process behind it but created during this boot: a crashed
+# pacman rather than a power cut. Not provable, so it is counted, and after
+# enough consecutive sightings the user is told to clean it up.
 CAU_STALE_LOCK_RUNS=3
 
 cau_track_stale_lock() {

src/lib/menu.sh

@@ -81,12 +81,23 @@ cau_ui_status() {
 		printf '\n  %s%s%s\n' "$CAU_C_YELLOW" \
 			"$(cau_msg "The last run reported a problem - see 'cachy-auto-update log'.")" \
 			"$CAU_C_RESET"
+	elif [[ $result == interrupted ]]; then
+		printf '\n  %s%s%s\n' "$CAU_C_YELLOW" \
+			"$(cau_msg "The last run was stopped before it finished.")" \
+			"$CAU_C_RESET"
 	fi
 	if [[ $reboot == 1 ]]; then
 		printf '\n  %s%s%s\n' "$CAU_C_YELLOW" \
 			"$(cau_msg "A restart is recommended to finish a kernel update.")" \
 			"$CAU_C_RESET"
 	fi
+
+	local held
+	held="$(cau_state_read held_back '')"
+	if [[ -n $held ]]; then
+		printf '\n  %s%s%s\n' "$CAU_C_YELLOW" \
+			"$(cau_msg "Held back: %s" "$held")" "$CAU_C_RESET"
+	fi
 }
 
 # cau_ui_status_conditions
@@ -145,31 +156,49 @@ cau_ui_menu() {
 		printf '  [q] %s\n' "$(cau_msg "Quit")"
 		printf '\n  > '
 
-		read -r choice || { printf '\n'; return 0; }
+		# One keypress, no Enter. -s keeps the raw character out of the
+		# display so the echo below is the only thing printed, and a failing
+		# read means EOF (Ctrl-D, or a script piping input) - that quits.
+		read -rsn1 choice || { printf '\n'; return 0; }
+		printf '%s\n' "$choice"
 
 		case "$choice" in
+			# The two switches flip and return straight to the menu, where the
+			# status block shows the result. Only a warning or an error holds
+			# the screen (see CAU_UI_NEEDS_ACK).
 			1)
+				CAU_UI_NEEDS_ACK=''
 				if [[ $CFG_ENABLED == yes ]]; then cau_do_disable; else cau_do_enable; fi
-				cau_pause
+				if [[ -n $CAU_UI_NEEDS_ACK ]]; then cau_pause; fi
 				;;
 			2)
+				CAU_UI_NEEDS_ACK=''
 				if [[ $CFG_NOTIFICATIONS == yes ]]; then
 					cau_do_notifications off
 				else
 					cau_do_notifications on
 				fi
-				cau_pause
+				if [[ -n $CAU_UI_NEEDS_ACK ]]; then cau_pause; fi
 				;;
 			3) cau_do_run --force; cau_pause ;;
 			4) cau_do_log; cau_pause ;;
 			5) cau_ui_status_conditions; cau_pause ;;
-			q|Q|'') return 0 ;;
+			q|Q) return 0 ;;
+			$'\e')
+				# Arrow keys and friends arrive as ESC [ X. Swallow the rest so
+				# one keypress does not redraw the menu three times. Escape is
+				# deliberately not a quit key: that would make a stray arrow
+				# key close the menu.
+				read -rsn2 -t 0.05 _ 2>/dev/null || true
+				;;
+			# Anything else, Enter included, just redraws.
 			*) ;;
 		esac
 	done
 }
 
 cau_pause() {
-	printf '\n  %s' "$(cau_msg "Press Enter to continue...")"
-	read -r _ || true
+	printf '\n  %s' "$(cau_msg "Press any key to continue...")"
+	read -rsn1 _ || true
+	printf '\n'
 }

src/lib/pkg_flatpak.sh

@@ -63,7 +63,10 @@ cau_flatpak_update() {
 		fi
 	done < <(cau_human_users)
 
-	if [[ $CFG_CLEAN_CACHE == yes ]]; then
+	# Unused runtimes are the Flatpak equivalent of orphaned packages - this is
+	# removal of installed software, not cache trimming, so it belongs behind
+	# RemoveOrphans rather than CleanCache.
+	if [[ $CFG_REMOVE_ORPHANS == yes ]]; then
 		cau_run_logged flatpak uninstall --system --unused --noninteractive --assumeyes || true
 	fi
 

src/lib/pkg_pacman.sh

@@ -10,15 +10,40 @@
 CAU_PACMAN_COUNT=0
 CAU_PACMAN_PENDING=''
 
+# Packages that had to be skipped so the rest of the upgrade could go through.
+CAU_PACMAN_HELD=''
+
 # Base flags for every unattended pacman invocation.
 cau_pacman_flags() {
-	printf '%s\n' --noconfirm --color never --noprogressbar --disable-download-timeout
+	printf '%s\n' --noconfirm --color never --disable-download-timeout
+
+	# A progress bar is worth having when somebody is watching a `run` from a
+	# terminal; in the timer's log it is only carriage-return noise.
+	[[ -n $CAU_INTERACTIVE ]] || printf '%s\n' --noprogressbar
+
 	local pkg
 	for pkg in $CFG_IGNORE_PKG; do
 		printf '%s\n' --ignore "$pkg"
 	done
 }
 
+# _cau_pacman_exec <logfile> <pacman args...>
+# Captures pacman's output for classification, and streams it as well when a
+# person is watching. Upgrading a few hundred packages takes minutes; without
+# this an interactive run shows one line and then nothing at all, which is
+# indistinguishable from a hang and invites someone to kill it mid-transaction.
+_cau_pacman_exec() {
+	local log="$1"
+	shift
+
+	if [[ -n $CAU_INTERACTIVE ]]; then
+		pacman "$@" 2>&1 | tee "$log"
+		return "${PIPESTATUS[0]}"
+	fi
+
+	pacman "$@" > "$log" 2>&1
+}
+
 # cau_pacman_pending
 # Fills CAU_PACMAN_PENDING and returns 1 when there is nothing to do.
 cau_pacman_pending() {
@@ -55,6 +80,8 @@ _cau_pacman_classify() {
 
 	if grep -qiE 'are in conflict|unresolvable package conflicts' "$log"; then
 		printf 'conflict\n'
+	elif grep -qiE 'could not satisfy dependencies|breaks dependency|unable to satisfy dependency' "$log"; then
+		printf 'dependency\n'
 	elif grep -qiE 'signature from .* is (unknown trust|marginal trust|invalid)|invalid or corrupted package \(PGP signature\)|key ".*" is unknown|keyring is not writable' "$log"; then
 		printf 'keyring\n'
 	elif grep -qiE 'exists in filesystem' "$log"; then
@@ -64,6 +91,25 @@ _cau_pacman_classify() {
 	fi
 }
 
+# _cau_pacman_blockers <logfile>
+# The packages standing in the way of an otherwise fine upgrade. pacman names
+# them in its dependency errors:
+#
+#   :: removing libperconaserverclient breaks dependency 'libperconaserverclient'
+#      required by heidisql-qt6-bin
+#   :: unable to satisfy dependency 'foo' required by bar
+#
+# In the first form the package being removed is the one to keep; in the second
+# it is the package that cannot be installed.
+_cau_pacman_blockers() {
+	local log="$1"
+
+	{
+		sed -nE "s/.*removing ([^ ]+) breaks dependency.*/\\1/p" "$log"
+		sed -nE "s/.*unable to satisfy dependency '[^']*' required by ([^ ]+).*/\\1/p" "$log"
+	} | grep -E '^[A-Za-z0-9@._+-]+$' | sort -u
+}
+
 # cau_pacman_update
 # Returns 0 on success (including "nothing to do"), 1 on a failure the user
 # needs to hear about. CAU_PACMAN_COUNT holds how many packages moved.
@@ -76,75 +122,106 @@ cau_pacman_update() {
 		return 0
 	fi
 
-	CAU_PACMAN_COUNT="$(grep -c . <<< "$CAU_PACMAN_PENDING")"
-	[[ $CAU_PACMAN_COUNT =~ ^[0-9]+$ ]] || CAU_PACMAN_COUNT=0
-	cau_info "Updating $CAU_PACMAN_COUNT repository package(s)"
+	if [[ -n $CAU_PACMAN_PENDING ]]; then
+		CAU_PACMAN_COUNT="$(grep -c . <<< "$CAU_PACMAN_PENDING")"
+		[[ $CAU_PACMAN_COUNT =~ ^[0-9]+$ ]] || CAU_PACMAN_COUNT=0
+		cau_info "Updating $CAU_PACMAN_COUNT repository package(s)"
+	else
+		# checkupdates is unavailable, so the list is unknown and pacman is
+		# asked to work it out itself.
+		CAU_PACMAN_COUNT=0
+		cau_info "Running a full system upgrade (pending list unavailable)"
+	fi
 
 	mapfile -t flags < <(cau_pacman_flags)
 	log="$(mktemp)" || return 1
 
-	if pacman -Syu "${flags[@]}" > "$log" 2>&1; then
-		cat "$log" >> "$CAU_RUNLOG" 2>/dev/null
-		rm -f "$log"
-		return 0
-	fi
+	# Recovery loop rather than a single retry: fixing one problem regularly
+	# uncovers the next (a conflict resolved into a dependency error, say).
+	# Each remedy is applied at most once, so this always terminates.
+	local -a extra=() blockers=() tried=()
+	local attempt=0 b
 
-	cat "$log" >> "$CAU_RUNLOG" 2>/dev/null
-	kind="$(_cau_pacman_classify "$log")"
-	cau_warn "pacman -Syu failed ($kind)"
-
-	case "$kind" in
-		keyring)
-			# A stale keyring is the one failure that is always safe to fix
-			# automatically, and it blocks everything else until it is.
-			cau_info "Refreshing keyrings and retrying"
-			local -a keyrings=()
-			pacman -Qq archlinux-keyring &> /dev/null && keyrings+=(archlinux-keyring)
-			pacman -Qq cachyos-keyring   &> /dev/null && keyrings+=(cachyos-keyring)
-			if (( ${#keyrings[@]} )); then
-				cau_run_logged pacman -Sy --noconfirm --color never "${keyrings[@]}" || true
-			fi
-			if pacman -Syu "${flags[@]}" > "$log" 2>&1; then
-				cat "$log" >> "$CAU_RUNLOG" 2>/dev/null
-				rm -f "$log"
-				return 0
-			fi
+	while true; do
+		if _cau_pacman_exec "$log" -Syu "${flags[@]}" "${extra[@]}"; then
 			cat "$log" >> "$CAU_RUNLOG" 2>/dev/null
-			;;
-
-		conflict)
-			# A package that has to replace another one. --noconfirm already
-			# answers "Replace X with Y?" affirmatively; what it declines is
-			# ":: X and Y are in conflict. Remove Y? [y/N]". --ask is pacman's
-			# question bitmask: 4 = CONFLICT_PKG, 16 = REMOVE_PKGS.
-			if [[ $CFG_RESOLVE_CONFLICTS != yes ]]; then
-				cau_error "Package conflict requires a decision (AutoResolveConflicts is off)"
-				rm -f "$log"
-				return 1
-			fi
-			cau_info "Resolving package conflicts automatically and retrying"
-			if pacman -Syu "${flags[@]}" --ask=20 > "$log" 2>&1; then
-				cat "$log" >> "$CAU_RUNLOG" 2>/dev/null
-				grep -E '^(removing|replacing) ' "$log" 2>/dev/null \
-					| while read -r line; do cau_info "  $line"; done
-				rm -f "$log"
-				return 0
-			fi
-			cat "$log" >> "$CAU_RUNLOG" 2>/dev/null
-			;;
-
-		filesystem)
-			# Untracked files in the way. Forcing --overwrite here could
-			# silently clobber something the user put there deliberately, so
-			# this one stays a human decision.
-			cau_error "Files on disk conflict with the update; manual review needed"
+			grep -E '^(removing|replacing) ' "$log" 2>/dev/null \
+				| while read -r line; do cau_info "  $line"; done
 			rm -f "$log"
-			return 1
-			;;
-	esac
+			return 0
+		fi
+
+		cat "$log" >> "$CAU_RUNLOG" 2>/dev/null
+		kind="$(_cau_pacman_classify "$log")"
+		cau_warn "pacman -Syu failed ($kind)"
+
+		if (( ++attempt > 3 )) || [[ " ${tried[*]} " == *" $kind "* ]]; then
+			break
+		fi
+		tried+=("$kind")
+
+		case "$kind" in
+			keyring)
+				# A stale keyring is the one failure that is always safe to fix
+				# automatically, and it blocks everything else until it is.
+				cau_info "Refreshing keyrings and retrying"
+				local -a keyrings=()
+				pacman -Qq archlinux-keyring &> /dev/null && keyrings+=(archlinux-keyring)
+				pacman -Qq cachyos-keyring   &> /dev/null && keyrings+=(cachyos-keyring)
+				if (( ${#keyrings[@]} )); then
+					cau_run_logged pacman -Sy --noconfirm --color never "${keyrings[@]}" || true
+				fi
+				;;
+
+			conflict)
+				# A package that has to replace another one. --noconfirm already
+				# answers "Replace X with Y?" affirmatively; what it declines is
+				# ":: X and Y are in conflict. Remove Y? [y/N]". --ask is pacman's
+				# question bitmask: 4 = CONFLICT_PKG, 16 = REMOVE_PKGS.
+				if [[ $CFG_RESOLVE_CONFLICTS != yes ]]; then
+					cau_error "Package conflict requires a decision (AutoResolveConflicts is off)"
+					break
+				fi
+				cau_info "Resolving package conflicts automatically and retrying"
+				extra+=(--ask=20)
+				;;
+
+			dependency)
+				# Something installed still depends on a package the repos want
+				# to drop or replace - almost always an AUR package that has not
+				# caught up yet. Nothing here can fix that, and it is not worth
+				# failing over: letting one stuck package block every other
+				# update indefinitely is far worse on an unattended machine.
+				# Hold the blockers back and upgrade everything else.
+				mapfile -t blockers < <(_cau_pacman_blockers "$log")
+				if (( ${#blockers[@]} == 0 )); then
+					cau_error "Dependency problem with no package to hold back"
+					break
+				fi
+				for b in "${blockers[@]}"; do
+					extra+=(--ignore "$b")
+				done
+				CAU_PACMAN_HELD="${blockers[*]}"
+				cau_warn "Holding back ${blockers[*]} and retrying without them"
+				;;
+
+			filesystem)
+				# Untracked files in the way. Forcing --overwrite here could
+				# silently clobber something the user put there deliberately, so
+				# this one stays a human decision.
+				cau_error "Files on disk conflict with the update; manual review needed"
+				break
+				;;
+
+			*)
+				break
+				;;
+		esac
+	done
 
 	rm -f "$log"
 	CAU_PACMAN_COUNT=0
+	CAU_PACMAN_HELD=''
 	return 1
 }
 
@@ -177,7 +254,14 @@ cau_pacman_cleanup() {
 	fi
 
 	if [[ $CFG_CLEAN_CACHE == yes ]] && cau_have paccache; then
-		cau_info "Trimming the package cache"
+		# paccache's dry run reports what a real run would reclaim; logging it
+		# first is the only way to tell afterwards whether trimming is doing
+		# anything, since the real run says little.
+		local reclaim
+		reclaim="$( { paccache -d --nocolor -k"$CFG_KEEP_OLD"; paccache -du --nocolor -k0; } 2>&1 \
+			| grep -oE 'disk space saved: [^)]*' | paste -sd', ' -)"
+
+		cau_info "Trimming the package cache (keeping $CFG_KEEP_OLD old version(s))${reclaim:+ - $reclaim}"
 		cau_run_logged paccache -r --nocolor -k"$CFG_KEEP_OLD"  || cau_warn "paccache -r failed"
 		cau_run_logged paccache -ru --nocolor -k0               || cau_warn "paccache -ru failed"
 	fi