Move the bar while packages are downloading

The progress entry sat at "0 von 123" for the whole download and only
started counting once pacman began unpacking - which on a domestic line is
most of the run spent looking like nothing was happening. What was being
watched for was the transaction, and the transaction had not started yet.

Downloading is now a step of its own, worth 30 of the bar against the
transaction's 40, counted the same way: one line per package out of
pacman's "foo-1.2-1-x86_64 downloading...". The database sync just before
it prints the identical shape with the suffix that would give it away
already stripped, so the tally starts only after the ":: Retrieving
packages..." header. Packages already in the cache never announce
themselves, so the step regularly ends short of its total and hands the
rest of its share over when unpacking begins.

The headline follows: "Updates werden heruntergeladen", then
"Systempakete werden aktualisiert".

README.md

@@ -214,11 +214,19 @@ Two things about how it is put together:
   the length of the update, holding the connection open and taking instructions
   on stdin. It needs **python-gobject**; without it there is simply no bar and
   nothing else changes.
-- The position inside the repository step comes from pacman's own
+- Downloading and unpacking are two separate steps on the bar. On a domestic
-  `(120/260) upgrading foo` lines. pacman's other `(n/m)` sequences — checking
+  line the download is the longer of the two, and calling the whole thing
-  keys, package integrity, loading files — each count to the same total, so
+  "installing" leaves the bar sitting at 4% for six minutes, which reads as a
-  only the transaction verbs are followed; otherwise the bar would reach the
+  hang rather than as progress.
-  end three times before the first package was unpacked.
+- Neither phase carries a counter on an unattended run, so both are counted a
+  line at a time — `foo-1.2-1-x86_64 downloading...` and `upgrading foo...`.
+  The database sync just before prints the same shape (` core downloading...`)
+  with the suffix that would give it away already stripped, so counting starts
+  only after pacman's `:: Retrieving packages...` header. pacman's other
+  `(n/m)` sequences — checking keys, package integrity, loading files — each
+  count to the same total, so only the transaction verbs are followed;
+  otherwise the bar would reach the end three times before the first package
+  was unpacked.
 
 This is Plasma's job interface. On a desktop that does not implement it the
 helper exits quietly and the ordinary notifications carry on as before.

doc/cachy-auto-update.1.scd

@@ -122,10 +122,18 @@ of the update and holds that connection open. It requires _python-gobject_.
 Where that is missing, or on a desktop with no job interface, there is no
 progress entry and nothing else is affected.
 
-The position within the repository step is read from pacman's own
+Fetching the packages and unpacking them are two steps rather than one. On a
-"(120/260) upgrading foo" output. Its other (n/m) sequences - checking keys,
+domestic line the download is the longer of the two, and a bar that called the
-package integrity, loading package files - each count up to the same total and
+whole thing "installing" would sit near its beginning for minutes at a time
-are deliberately ignored.
+looking stuck.
+
+Neither phase gets a counter from pacman on an unattended run, so both are
+counted here, a line at a time: "foo-1.2-1-x86_64 downloading..." for the
+first, "upgrading foo..." for the second. Packages already in the cache are
+never announced, so the download step regularly ends short of its total and
+gives up the rest of its share when unpacking begins. pacman's other (n/m)
+sequences - checking keys, package integrity, loading package files - each
+count up to the same total and are deliberately ignored.
 
 # HOW LONG NOTIFICATIONS STAY
 

po/cachy-auto-update.pot

@@ -234,6 +234,9 @@ msgstr ""
 #. looking for it - it appears next to whatever they were doing - so each one
 #. says outright that this is an update running, rather than naming the kind of
 #. package on its own.
+msgid "Downloading updates"
+msgstr ""
+
 msgid "Updating system packages"
 msgstr ""
 

po/de.po

@@ -235,6 +235,9 @@ msgstr "Ohne Befehl wird ein interaktives Menü angezeigt."
 #. looking for it - it appears next to whatever they were doing - so each one
 #. says outright that this is an update running, rather than naming the kind of
 #. package on its own.
+msgid "Downloading updates"
+msgstr "Updates werden heruntergeladen"
+
 msgid "Updating system packages"
 msgstr "Systempakete werden aktualisiert"
 

src/cachy-auto-update-run

@@ -185,7 +185,7 @@ failed=0
 # Open the desktop's progress bar, told up front which steps this run will
 # perform. Only those count towards the bar, so a machine with no Flatpaks
 # does not sit at 85% for the last second of the run.
-progress_steps=(repo)
+progress_steps=(download repo)
 [[ $CFG_AUR == yes ]]      && progress_steps+=(aur)
 [[ $CFG_FLATPAK == yes ]]  && progress_steps+=(flatpak)
 [[ $CFG_APPIMAGE == yes ]] && progress_steps+=(appimage)

src/lib/pkg_pacman.sh

@@ -32,11 +32,34 @@ cau_pacman_flags() {
 # precisely so pacman's output stays parseable.
 CAU_PACMAN_OP_RE='^(\([[:space:]]*[0-9]+/[0-9]+\) )?(upgrading|installing|reinstalling|downgrading|removing) [^[:space:]]+'
 
-# _cau_pacman_progress_watch <logfile>
+# Before any of that, everything has to be fetched, and on a domestic line
-# Feeds the desktop's progress bar by watching pacman work.
+# that is the longer half of the run: two hundred packages take minutes to
+# arrive and seconds to unpack. pacman prints one line per package while it
+# does it,
 #
-# pacman announces each package twice over, in one of two shapes, and which one
+#    glibc-2.44+r24+g16be1518495f-1-x86_64_v3 downloading...
-# depends on a flag this program sets itself:
+#
+# and nothing else - no counter, no total - so the position here is counted the
+# same way the transaction is.
+#
+# The database sync a few lines earlier prints the very same shape (" core
+# downloading..."), and pacman strips the suffix that would tell a database
+# from a package, so the count begins only after the header that separates the
+# two phases.
+CAU_PACMAN_DL_AWK='
+	/^:: Retrieving packages/ { retrieving = 1; next }
+	retrieving && / downloading\.\.\.$/ { n++; name = $1 }
+	END { print n + 0, name }'
+
+# _cau_pacman_progress_watch <logfile>
+# Feeds the desktop's progress bar by watching pacman work, through both of the
+# phases a pacman run has: first everything is fetched, then everything is
+# unpacked. They are two steps on the bar rather than one, because they are two
+# steps to sit through - a run that has been "installing updates" at 4% for six
+# minutes has not hung, it is still downloading, and the bar should say so.
+#
+# In the transaction, pacman announces each package twice over, in one of two
+# shapes, and which one depends on a flag this program sets itself:
 #
 #   upgrading glibc...              with --noprogressbar, i.e. every timer run
 #   ( 12/218) upgrading glibc [##]  with the bar, i.e. an interactive `run`
@@ -60,6 +83,7 @@ CAU_PACMAN_OP_RE='^(\([[:space:]]*[0-9]+/[0-9]+\) )?(upgrading|installing|reinst
 _cau_pacman_progress_watch() {
 	local log="$1"
 	local total="${CAU_PACMAN_COUNT:-0}" announced processed line pkg last=''
+	local phase=download fetched shown=''
 
 	while :; do
 		sleep 1
@@ -69,7 +93,30 @@ _cau_pacman_progress_watch() {
 		[[ $announced =~ ^[0-9]+$ ]] && (( announced > 0 )) && total="$announced"
 
 		line="$(grep -aoE "$CAU_PACMAN_OP_RE" "$log" 2>/dev/null | tail -n1)"
-		[[ -n $line ]] || continue
+
+		# Nothing unpacked yet, so this is still the download - or the database
+		# sync ahead of it, which the awk above declines to count.
+		if [[ -z $line ]]; then
+			read -r fetched pkg < <(awk "$CAU_PACMAN_DL_AWK" "$log" 2>/dev/null)
+			[[ $fetched =~ ^[0-9]+$ ]] && (( fetched > 0 )) || continue
+			[[ $fetched != "$shown" ]] || continue
+			shown="$fetched"
+
+			cau_progress_item "$fetched" "$total"
+			# Down to the bare name, as the transaction reports it: the file
+			# pacman names here carries version, release and architecture.
+			cau_progress_detail "Package" "${pkg%-*-*-*}"
+			continue
+		fi
+
+		# The first package being unpacked ends the download step. Its share of
+		# the bar is given up wherever it had got to - packages already in the
+		# cache are fetched in no time at all and never print a line, so the
+		# tally regularly stops short of the total it was promised.
+		if [[ $phase == download ]]; then
+			phase=install
+			cau_progress_step repo "Updating system packages" "$total"
+		fi
 
 		# Nothing new since the last look. Checked before the counting grep
 		# because on a large upgrade this loop spends most of its life here.
@@ -198,7 +245,9 @@ cau_pacman_update() {
 	local log kind
 	local -a flags
 
-	cau_progress_step repo "Updating system packages"
+	# The download comes first and the watcher moves on to the repo step once
+	# pacman starts unpacking.
+	cau_progress_step download "Downloading updates"
 
 	if ! cau_pacman_pending; then
 		cau_info "No repository updates pending"

src/lib/progress.sh

@@ -26,11 +26,12 @@ CAU_PROGRESS_FIFOS=()
 CAU_PROGRESS_LOCALES=()
 
 # What each step is worth on the bar. Rough shares of a typical run rather than
-# anything measured: repositories dominate, the cleanup is a rounding error.
+# anything measured: the repositories dominate - fetching them and unpacking
+# them about equally, on a domestic line - and the cleanup is a rounding error.
 # They do not have to add up to 100 - only the steps a given run will actually
 # perform are counted, and the total is normalised against those.
 declare -A CAU_PROGRESS_WEIGHTS=(
-	[repo]=70 [aur]=15 [flatpak]=10 [appimage]=3 [cleanup]=2
+	[download]=30 [repo]=40 [aur]=15 [flatpak]=10 [appimage]=3 [cleanup]=2
 )
 
 CAU_PROGRESS_PLAN=()