Say an update is running before it starts, not only after

Asked what happens if someone shuts the laptop down mid-update, the honest
answer turned out to be poor. The inhibitor does stop them - logind refuses and
falls back to org.freedesktop.login1.power-off-ignore-inhibit, which is
auth_admin_keep - so the desktop answers with an administrator password prompt
reading "Power off the system while an application is inhibiting this". systemd
ships that string untranslated, it never mentions updates, and no KDE catalog
contains any shutdown-blocked text at all. Only systemctl names the reason.

So a notification now goes out before the transaction, asking for the machine to
be left on. Notifications gained a tag: the result replaces the start message in
place rather than stacking a second, contradictory bubble beside it.

Tagged notifications also carry a queue flag. "An update is starting" is only
meaningful while somebody is looking, so unlike the result it is not spooled for
delivery at next login.

Makefile

@@ -8,7 +8,7 @@
 # Overridable so a packager can pass the version it is actually building
 # (`make VERSION=$pkgver`). The literal below is the fallback for builds
 # straight from a checkout, and is what a release tag has to carry.
-VERSION      ?= 1.0.7
+VERSION      ?= 1.0.9
 
 PREFIX       ?= /usr
 DESTDIR      ?=

README.md

@@ -123,6 +123,48 @@ A leftover `db.lck` from a crashed transaction is never deleted automatically
 guessing wrong there corrupts a live transaction. After it has been seen
 unheld on several consecutive runs, you get a notification instead.
 
+## What if the machine is switched off mid-update
+
+Three layers, in order of how much they can actually promise:
+
+**A notification goes out before the transaction starts** — "Installing
+updates, please leave the computer switched on until this is done" — and is
+replaced in place by the result when the run finishes, so it costs one bubble
+rather than two. This exists because of what the next paragraph does *not* do.
+
+**Suspend and a normal shutdown are blocked.** The run holds a
+`systemd-inhibit --what=sleep:shutdown --mode=block` lock, so closing the lid or
+picking "Shut down" cannot interrupt a transaction. Be aware of what that looks
+like, though: logind refuses the request and requires the polkit action
+`org.freedesktop.login1.power-off-ignore-inhibit`, which is `auth_admin_keep`.
+The desktop therefore answers a shutdown attempt with an **administrator
+password prompt** reading *"Power off the system while an application is
+inhibiting this"* — a string systemd ships untranslated, and one that never
+mentions updates. No KDE dialog explains the situation. Only `systemctl
+poweroff` in a terminal names the reason. That prompt is exactly why the
+notification above is on by default.
+
+**A hard power-off cannot be prevented by anything.** Holding the power button
+or pulling the plug cuts power in firmware. What limits the damage is that
+pacman's commit phase is short (about a minute even for a 200-package upgrade)
+and that most of a run is downloading, where an interruption costs nothing but
+a partial file.
+
+**The next run repairs it.** A `db.lck` left behind is detected and removed —
+but only when it is *provably* dead, meaning it is older than the current boot,
+so no process that could hold it still exists. The interrupted upgrade is then
+simply run again; pacman reinstalls anything that was caught half-written. A
+lock that is merely unheld within the same boot is never removed, only
+reported, because there the guess could be wrong.
+
+This last part matters more than it sounds: without it, a single power cut
+during an update would leave a lock file that makes every future run defer,
+and the machine would stop updating silently and permanently.
+
+On a Btrfs system with `snapper` and `snap-pac` — the CachyOS default — every
+pacman transaction is bracketed by a pre and post snapshot, so a genuinely
+broken upgrade can still be rolled back with `snapper rollback`.
+
 ## Configuration
 
 `/etc/cachy-auto-update/cachy-auto-update.conf`, one `Key=Value` per line, every

doc/cachy-auto-update.1.scd

@@ -100,6 +100,39 @@ running are skipped.
 The machine is never restarted automatically. When a kernel update makes a
 restart necessary, a notification says so.
 
+# INTERRUPTED UPDATES
+
+Before a transaction starts, a notification says an update is running and asks
+for the machine to be left on. It is replaced in place by the result once the
+run finishes.
+
+While a transaction is running, *cachy-auto-update* holds a
+*systemd-inhibit*(1) lock on _sleep_ and _shutdown_ in blocking mode, so a
+suspend, a lid close or a normal shutdown request cannot cut it short.
+
+What a user sees when they try anyway is worth knowing: logind refuses the
+request and falls back to the polkit action
+_org.freedesktop.login1.power-off-ignore-inhibit_, which is _auth_admin_keep_,
+so the desktop presents an administrator password prompt reading "Power off the
+system while an application is inhibiting this". That string is shipped
+untranslated by systemd and does not mention updates, and no KDE dialog
+explains the situation either - only *systemctl*(1) names the inhibitor and its
+reason. Hence the notification.
+
+A hard power-off - holding the power button, or losing mains power - is not
+preventable. On the next run a leftover _/var/lib/pacman/db.lck_ is removed if
+it is older than the current boot, since no process able to hold it can still
+exist; the upgrade is then repeated and pacman reinstalls whatever was caught
+half-written. A lock file that is unheld but was created during the current
+boot is reported rather than removed, because there is no way to prove it is
+abandoned.
+
+Without that recovery a single power cut would leave a lock that makes every
+subsequent run defer, silently stopping updates for good.
+
+On Btrfs with *snapper*(8) and *snap-pac*, each pacman transaction is bracketed
+by a pre and post snapshot, so a broken upgrade remains rollbackable.
+
 # FILES
 
 _/etc/cachy-auto-update/cachy-auto-update.conf_

po/cachy-auto-update.pot

@@ -257,3 +257,15 @@ msgstr ""
 
 msgid "The last run was stopped before it finished."
 msgstr ""
+
+msgid "Finishing an interrupted update"
+msgstr ""
+
+msgid "The last update was cut short, most likely because the machine was switched off. It is being finished now."
+msgstr ""
+
+msgid "Installing updates"
+msgstr ""
+
+msgid "%d packages are being updated. Please leave the computer switched on until this is done."
+msgstr ""

po/de.po

@@ -258,3 +258,16 @@ msgstr "Zurückgehalten: %s"
 
 msgid "The last run was stopped before it finished."
 msgstr "Der letzte Lauf wurde abgebrochen, bevor er fertig war."
+
+msgid "Finishing an interrupted update"
+msgstr "Abgebrochenes Update wird beendet"
+
+msgid "The last update was cut short, most likely because the machine was switched off. It is being finished now."
+msgstr "Das letzte Update wurde unterbrochen, vermutlich weil der Rechner ausgeschaltet wurde. Es wird jetzt zu Ende geführt."
+
+msgid "Installing updates"
+msgstr "Updates werden installiert"
+
+#, c-format
+msgid "%d packages are being updated. Please leave the computer switched on until this is done."
+msgstr "%d Pakete werden gerade aktualisiert. Bitte den Rechner so lange eingeschaltet lassen."

res/config/cachy-auto-update.conf

@@ -95,6 +95,12 @@ RemoveOrphans=no
 # Notification detail
 # ---------------------------------------------------------------------------
 
+# Say that an update has started. Worth keeping on: while one runs, a shutdown
+# request is refused and the desktop answers with an untranslated polkit
+# password prompt that never mentions updates. This message is replaced by the
+# result when the run finishes, so it costs one bubble, not two.
+NotifyOnStart=yes
+
 # Say something after a successful update. Nothing is ever shown when there
 # was nothing to do.
 NotifyOnSuccess=yes

src/cachy-auto-update-run

@@ -117,6 +117,15 @@ if (( ! FORCE )); then
 	fi
 fi
 
+# A lock left behind by a power cut during a previous update. It is cleared
+# before the busy check, because otherwise every future run would defer on it
+# forever and the machine would quietly stop updating.
+if cau_recover_stale_lock; then
+	cau_notify normal \
+		"Finishing an interrupted update" \
+		"The last update was cut short, most likely because the machine was switched off. It is being finished now."
+fi
+
 # This one is checked even with --force: proceeding anyway would just hand the
 # user a lock error instead of doing anything useful.
 CAU_SKIP_REASON=''
@@ -204,7 +213,7 @@ cau_state_write last_counts \
 if (( failed )); then
 	cau_state_write last_result failed
 	cau_error "Update run finished with errors"
-	[[ $CFG_NOTIFY_ERROR == yes ]] && cau_notify critical \
+	[[ $CFG_NOTIFY_ERROR == yes ]] && cau_notify_tagged run yes critical \
 		"Update failed" \
 		"Something went wrong while updating. Run 'cachy-auto-update log' for details."
 else
@@ -213,7 +222,7 @@ else
 	cau_info "Update run finished successfully ($total item(s) updated)"
 
 	if (( total > 0 )) && [[ $CFG_NOTIFY_SUCCESS == yes ]]; then
-		cau_notify low "System updated" "%d updates were installed." "$total"
+		cau_notify_tagged run yes low "System updated" "%d updates were installed." "$total"
 	fi
 fi
 

src/lib/config.sh

@@ -98,6 +98,7 @@ cau_config_load() {
 	CFG_RESOLVE_CONFLICTS=no;     cau_config_bool AutoResolveConflicts yes && CFG_RESOLVE_CONFLICTS=yes
 	CFG_CLEAN_CACHE=no;           cau_config_bool CleanCache           yes && CFG_CLEAN_CACHE=yes
 	CFG_REMOVE_ORPHANS=no;        cau_config_bool RemoveOrphans        no  && CFG_REMOVE_ORPHANS=yes
+	CFG_NOTIFY_START=no;          cau_config_bool NotifyOnStart        yes && CFG_NOTIFY_START=yes
 	CFG_NOTIFY_SUCCESS=no;        cau_config_bool NotifyOnSuccess      yes && CFG_NOTIFY_SUCCESS=yes
 	CFG_NOTIFY_ERROR=no;          cau_config_bool NotifyOnError        yes && CFG_NOTIFY_ERROR=yes
 	CFG_NOTIFY_REBOOT=no;         cau_config_bool NotifyReboot         yes && CFG_NOTIFY_REBOOT=yes

src/lib/locks.sh

@@ -60,11 +60,55 @@ cau_package_manager_busy() {
 	return 1
 }
 
+# cau_pacman_lock_is_stale
+# True only when the lock provably cannot belong to anything alive.
+#
+# The rigorous test is the boot time: no process that existed before the
+# current boot can still be running, so a db.lck older than boot is abandoned
+# by definition - which is exactly what a power cut during an update leaves
+# behind. A lock that is merely unheld *within* this boot is not provable in
+# the same way, so it is only reported (see cau_track_stale_lock) and never
+# removed; guessing wrong there would corrupt a live transaction.
+#
+# The fuser check is kept as a second condition purely to survive a backwards
+# clock jump making a live lock look pre-boot.
+cau_pacman_lock_is_stale() {
+	local boot lock
+
+	[[ -e $CAU_PACMAN_LOCK ]] || return 1
+
+	boot="$(awk '/^btime /{print $2}' /proc/stat 2>/dev/null)"
+	[[ $boot =~ ^[0-9]+$ ]] || return 1
+
+	lock="$(stat -c %Y "$CAU_PACMAN_LOCK" 2>/dev/null)" || return 1
+	[[ $lock =~ ^[0-9]+$ ]] || return 1
+
+	(( lock < boot )) || return 1
+	[[ -z "$(cau_pacman_lock_holder)" ]]
+}
+
+# cau_recover_stale_lock
+# Clears a provably abandoned lock so an interrupted update can be finished on
+# the next run. Without this, one power cut during an update stops every future
+# update permanently and silently - the worst possible outcome for a machine
+# nobody is watching.
+cau_recover_stale_lock() {
+	cau_pacman_lock_is_stale || return 1
+
+	cau_warn "Found a pacman lock older than this boot - an update was cut short"
+	rm -f "$CAU_PACMAN_LOCK" 2>/dev/null || {
+		cau_error "Could not remove the stale pacman lock"
+		return 1
+	}
+	cau_state_clear stale_lock_count
+	cau_info "Stale lock removed; the interrupted update will be finished now"
+	return 0
+}
+
 # cau_track_stale_lock
-# A db.lck with no process behind it is left over from a crashed transaction.
-# Removing it automatically would be reckless - if the guess is wrong it
-# corrupts a live transaction - so instead it is counted, and after enough
-# consecutive sightings the user is told to clean it up.
+# A db.lck with no process behind it but created during this boot: a crashed
+# pacman rather than a power cut. Not provable, so it is counted, and after
+# enough consecutive sightings the user is told to clean it up.
 CAU_STALE_LOCK_RUNS=3
 
 cau_track_stale_lock() {

src/lib/notify.sh

@@ -19,10 +19,23 @@ CAU_NOTIFY_QUEUE_MAX=20
 # Never fails: a machine without libnotify, or with nobody logged in, is a
 # normal state, not an error.
 cau_notify() {
-	local urgency="$1" title="$2" body="$3"
-	shift 3
+	cau_notify_tagged '' yes "$@"
+}
+
+# cau_notify_tagged <tag> <queue: yes|no> <urgency> <title> <body> [args...]
+#
+# A tag makes this notification replace the previous one carrying the same tag
+# rather than stacking a second bubble beside it - that is what turns
+# "installing updates" into "system updated" in place instead of leaving two
+# messages that contradict each other.
+#
+# queue=no is for messages that only mean anything while somebody is looking.
+# Telling a user at next login that an update started an hour ago is noise.
+cau_notify_tagged() {
+	local tag="$1" queue="$2" urgency="$3" title="$4" body="$5"
+	shift 5
 	local -a args=("$@")
-	local delivered=0 user uid locale t b
+	local delivered=0 user uid locale t b prev newid
 
 	[[ $CFG_NOTIFICATIONS == yes ]] || return 0
 
@@ -34,17 +47,28 @@ cau_notify() {
 		t="$(cau_msg_in "$locale" "$title")"
 		b="$(cau_msg_in "$locale" "$body" "${args[@]}")"
 
-		if cau_as_user "$user" "$uid" notify-send \
+		prev=0
+		if [[ -n $tag ]]; then
+			prev="$(cau_state_read "notify_id_${tag}_${user}" 0)"
+			[[ $prev =~ ^[0-9]+$ ]] || prev=0
+		fi
+
+		if newid="$(cau_as_user "$user" "$uid" notify-send \
 			--app-name="$CAU_PRETTY" \
 			--icon="$CAU_NOTIFY_ICON" \
 			--urgency="$urgency" \
-			-- "$t" "$b" 2>/dev/null
+			--print-id --replace-id="$prev" \
+			-- "$t" "$b" 2>/dev/null)"
 		then
 			delivered=1
+			if [[ -n $tag && $newid =~ ^[0-9]+$ ]]; then
+				cau_state_write "notify_id_${tag}_${user}" "$newid"
+			fi
 		fi
 	done < <(cau_active_session_users)
 
 	(( delivered )) && return 0
+	[[ $queue == yes ]] || return 0
 
 	cau_notify_enqueue "$urgency" "$title" "$body" "${args[@]}"
 }

src/lib/pkg_pacman.sh

@@ -133,6 +133,20 @@ cau_pacman_update() {
 		cau_info "Running a full system upgrade (pending list unavailable)"
 	fi
 
+	# Say so before the transaction starts, not after it finishes.
+	#
+	# While an upgrade runs, a shutdown request is refused by logind and the
+	# desktop answers with a polkit password prompt reading "Power off the
+	# system while an application is inhibiting this" - which never mentions
+	# updates and, on a German system, is not even translated. Somebody who was
+	# simply told beforehand does not end up staring at that.
+	if [[ $CFG_NOTIFY_START == yes ]]; then
+		cau_notify_tagged run no normal \
+			"Installing updates" \
+			"%d packages are being updated. Please leave the computer switched on until this is done." \
+			"${CAU_PACMAN_COUNT:-0}"
+	fi
+
 	mapfile -t flags < <(cau_pacman_flags)
 	log="$(mktemp)" || return 1