feat(discovery): enumerate the apps installed on the Shield (t57)

The live complaint ("it doesn't find all the apps") came from the curated
list: only four packages were ever offered, so anything the user had
installed elsewhere was invisible. The device knows its own apps.

ha_adb_common/installed_apps.py parses the two queries the sweep makes:

* `pm list packages -3` — every third-party (user-installed) package;
* `cmd package resolve-activity --brief -c android.intent.category.HOME`
  — the ACTIVE launcher, a system app that is nonetheless what the user
  opens, so it is appended rather than filtered away.

System apps are otherwise not enumerated (a Shield ships a few hundred and
they are not launchable apps); the parsers are tolerant of the noise a real
adb shell can prepend and of duplicate lines.

ShieldConnectionManager gains the sweep:
`async_refresh_installed_apps(force=False)` returns a cached
`[{package, name, icon}]` table, and `installed_apps()` is a pure read that
never issues a shell — so a state poll costs nothing.

Naming stays on the one path: the sweep calls `_dump_label`, the same
`pm dump` + parse + 24 h cache the entity path uses, now extracted so the
two cannot drift, including the t57 UNIT 0 rule that a fetch that returned
nothing is NOT cached. Labels are fetched APP_SWEEP_CONCURRENCY (4) at a
time with APP_SWEEP_SPACING (50 ms), so a typical Shield finishes in
seconds rather than the minutes a one-per-watchdog-cycle drain would take
while the socket still sees a trickle.

Failure is a normal outcome: no poller, a disconnected transport or a mute
shell stores an EMPTY table with a stamp and returns [], so nothing retries
in a loop and every surface degrades to "no discovered apps".

Icons: `pm dump` does NOT expose an app's icon resource — an extracted
APK's icon is a bitmap only its own code resolves — so this is honestly a
heuristic: APP_ICON_FALLBACKS (a curated package -> mdi: map) with
APP_ICON_GENERIC (mdi:television) for anything unknown, and always a string.

custom_components/nvidia_shield/connection.py

@@ -36,8 +36,13 @@ from .const import (
     ADB_RECONNECT_BACKOFF_CAP,
     ADB_RECONNECT_BACKOFF_START,
     ADB_RECONNECT_JITTER,
+    APP_ICON_FALLBACKS,
+    APP_ICON_GENERIC,
     APP_LABEL_EXIT_MARKER,
     APP_LABEL_TTL,
+    APP_LIST_TTL,
+    APP_SWEEP_CONCURRENCY,
+    APP_SWEEP_SPACING,
     CONF_LOGCAT_EVENTS,
     LOGCAT_SUPPRESS_WINDOW,
     STATE_STORE_TTL,
@@ -46,6 +51,9 @@ from .const import (
 from .ha_adb_common import (
     MediaSessionInfo,
     foreground_from_logcat,
+    icon_for_package,
+    installed_packages,
+    is_launchable,
     parse_application_label,
     parse_audio_playback,
 )
@@ -216,6 +224,13 @@ class ShieldConnectionManager:
         # t55: packages whose label fetch is queued or in flight — a render
         # during the fetch must not enqueue the same package twice.
         self._app_label_inflight: set[str] = set()
+        # t57: the discovered installed-apps table — (rows, stamp), or an empty
+        # row list with a stamp when ADB cannot answer. APP_LIST_TTL, refreshed
+        # on demand; never rebuilt on a state poll.
+        self._installed_apps: tuple[list[dict], float] | None = None
+        #: t57: the in-flight sweep, so concurrent callers share one sweep
+        #: instead of stacking a second burst on the same connection.
+        self._apps_sweep: asyncio.Task | None = None
         # P5-3 (C-STREAM): the dedicated logcat line-stream consumer.
         # NEVER shares the poller's ADB socket — its own device object and
         # connection; CONF_LOGCAT_EVENTS gates it (default off) and the
@@ -636,70 +651,71 @@ class ShieldConnectionManager:
         except Exception as err:
             _LOGGER.debug("protocol device info unavailable: %s", err)
 
+    async def _dump_label(self, package: str) -> str | None:
+        """One `pm dump` label read: the label, or None for a FAILED fetch.
+
+        The ONE place the dump is issued, shared by the on-demand path
+        (:meth:`_fetch_app_label`) and the discovery sweep, so the t57 UNIT 0
+        distinction — "the command ran and found no label" versus "the
+        transport delivered nothing" — cannot drift between them. A successful
+        fetch is also cached here, so both paths benefit from the 24 h cache and
+        neither double-stamps.
+
+        Returns the label (possibly "") when the dump RAN, and None when it did
+        not run or delivered no output at all.
+        """
+        poller = self._poller
+        if poller is None or not self.connected:
+            _LOGGER.debug(
+                "app label for %s not fetched (ADB not connected) — will "
+                "retry on the next render",
+                package,
+            )
+            return None
+        try:
+            import shlex
+
+            dump = await poller.shell(
+                f"pm dump {shlex.quote(package)} 2>/dev/null | head -300; "
+                f"echo {APP_LABEL_EXIT_MARKER}$?"
+            )
+        except Exception as err:
+            _LOGGER.debug(
+                "app label fetch failed for %s (%s) — will retry on the "
+                "next render",
+                package,
+                err,
+            )
+            return None
+        if dump is None or not dump.strip():
+            # the transport never delivered output: a FAILED fetch, not an
+            # answer. Caching here is the t55 defect this repairs.
+            _LOGGER.debug(
+                "app label fetch for %s returned no output — will retry on "
+                "the next render",
+                package,
+            )
+            return None
+        label = parse_application_label(dump)
+        # F1: label-less pm dumps negatively cache too (empty label) so a
+        # later cycle performs zero additional pm dumps for the package.
+        self.cache_app_label(package, label)
+        return label
+
     async def _fetch_app_label(self, package: str) -> None:
-        """One `pm dump` label fetch; stamps ONLY a fetch that really RAN (t57).
+        """One on-demand `pm dump` label fetch; stamps ONLY a fetch that RAN.
 
-        The distinction this method exists for (t59 F1): a FAILED fetch and a
+        The on-demand half of the label path (one package per watchdog cycle,
-        genuinely label-less package are not the same thing, and conflating
+        driven by an entity render). The dump itself — including the t57 UNIT 0
-        them is what kept ``app_name`` on the raw package id forever.
+        distinction between a FAILED fetch and a genuinely label-less package —
-
+        lives in :meth:`_dump_label`, so the sweep and this path cannot drift.
-        * the dump ran and carries an ``application-label`` → cache the label;
-        * the dump ran and carries none → cache the empty string as a
-          definitive negative (t24 F1: no per-heartbeat round-trip tax for a
-          label-less package);
-        * the fetch did NOT run → cache NOTHING, and the next render
-          re-enqueues. ``ShieldAdbPoller.shell`` answers ``None`` for both
-          shapes of "did not run" — no connected device, and a command that
-          raised — so ``None``/blank output is treated as a failure rather
-          than being read as "the dump ran and found no label". Treating it
-          as the latter stamped a 24 h definitive negative after ONE transient
-          failure, and a later good dump could never recover the label.
-
-        The dump also carries an exit marker (``; echo __NSH_EXIT:$?``), which
-        is what keeps the third branch from swallowing a REAL empty answer: if
-        the command ran and produced nothing, the marker is still there, so
-        the negative cache happens (zero further round trips). If the output
-        is blank the command never delivered anything, and we retry.
         """
         self._app_label_inflight.add(package)
         try:
-            poller = self._poller
+            await self._dump_label(package)
-            if poller is None or not self.connected:
-                _LOGGER.debug(
-                    "app label for %s not fetched (ADB not connected) — will "
-                    "retry on the next render",
-                    package,
-                )
-                return
-            try:
-                import shlex
-
-                dump = await poller.shell(
-                    f"pm dump {shlex.quote(package)} 2>/dev/null | head -300; "
-                    f"echo {APP_LABEL_EXIT_MARKER}$?"
-                )
-            except Exception as err:
-                _LOGGER.debug(
-                    "app label fetch failed for %s (%s) — will retry on the "
-                    "next render",
-                    package,
-                    err,
-                )
-                return
-            if dump is None or not dump.strip():
-                # the transport never delivered output: a FAILED fetch, not an
-                # answer. Caching here is the t55 defect this repairs.
-                _LOGGER.debug(
-                    "app label fetch for %s returned no output — will retry on "
-                    "the next render",
-                    package,
-                )
-                return
-            # F1: label-less pm dumps negatively cache too (empty label) so a
-            # later cycle performs zero additional pm dumps for the package.
-            self.cache_app_label(package, parse_application_label(dump))
         finally:
             self._app_label_inflight.discard(package)
+
     def _on_logcat_app_start(self, package: str) -> None:
         """Stamp FIELD_FOREGROUND instantly from a parsed START (P5-3).
 
@@ -851,6 +867,164 @@ class ShieldConnectionManager:
             self._logcat_task.cancel()
             self._logcat_task = None
 
+    # ── t57: installed-app discovery ───────────────────────────────────────
+
+    def installed_apps(self) -> list[dict]:
+        """The discovered apps, ``[{package, name, icon}]``, cached (t57).
+
+        Returns the cached table whenever it is fresh, and an EMPTY list when
+        there is nothing usable yet (ADB disabled, not connected, or the first
+        sweep still running). A caller that wants the table filled calls
+        :meth:`async_refresh_installed_apps`; this accessor never blocks and
+        never issues a shell itself, so a state poll stays free of discovery
+        work.
+        """
+        if self._installed_apps is None:
+            return []
+        rows, stamp = self._installed_apps
+        if self._clock() - stamp > APP_LIST_TTL:
+            return []  # stale — a refresh repopulates it, the poll does not
+        return [dict(row) for row in rows]
+
+    @property
+    def installed_apps_stale(self) -> bool:
+        """True when no fresh table is available (including "never swept")."""
+        if self._installed_apps is None:
+            return True
+        _rows, stamp = self._installed_apps
+        return self._clock() - stamp > APP_LIST_TTL
+
+    async def async_refresh_installed_apps(self, force: bool = False) -> list[dict]:
+        """(Re)build the installed-app table over ADB, bounded and shared (t57).
+
+        One sweep at a time: concurrent callers await the same task rather than
+        starting a second burst, and a fresh table is returned as-is unless
+        ``force``. The sweep is deliberately NOT part of the watchdog: it is a
+        handful of shells on demand, not a per-cycle cost.
+
+        Failure is normal and handled without noise — no poller, a disconnected
+        transport, or a shell that answers nothing stores an EMPTY table with a
+        stamp, so the caller degrades to "no discovered apps" and nothing
+        retries in a loop. The next explicit refresh (or the TTL) tries again.
+        """
+        if not force and not self.installed_apps_stale:
+            return self.installed_apps()
+        if self._apps_sweep is not None and not self._apps_sweep.done():
+            await asyncio.shield(self._apps_sweep)
+            return self.installed_apps()
+        task = self._create_task(self._sweep_installed_apps(), "apps_sweep")
+        self._apps_sweep = task
+        try:
+            rows = await task
+        finally:
+            if self._apps_sweep is task:
+                self._apps_sweep = None
+        return [dict(row) for row in rows]
+
+    async def _sweep_installed_apps(self) -> list[dict]:
+        """The sweep itself: list, label, icon — then stamp the cache."""
+        poller = self._poller
+        if poller is None or not self.connected:
+            _LOGGER.debug(
+                "installed-app discovery skipped (ADB unavailable) — the "
+                "options page and the app list stay on what is already known"
+            )
+            self._installed_apps = ([], self._clock())
+            return []
+        try:
+            user_output = await poller.shell("pm list packages -3")
+            launcher_output = await poller.shell(
+                "cmd package resolve-activity --brief -c "
+                "android.intent.category.HOME"
+            )
+        except Exception as err:  # pragma: no cover - shell swallows its own
+            _LOGGER.debug("installed-app discovery failed: %s", err)
+            self._installed_apps = ([], self._clock())
+            return []
+        if user_output is None and launcher_output is None:
+            # the transport delivered nothing at all: no list is an ANSWER
+            _LOGGER.debug(
+                "installed-app discovery got no output — will try again on the "
+                "next refresh"
+            )
+            self._installed_apps = ([], self._clock())
+            return []
+
+        packages = [p for p in installed_packages(user_output, launcher_output)
+                    if is_launchable(p)]
+        labels = await self._sweep_labels(packages)
+        rows = [
+            {
+                "package": package,
+                "name": labels.get(package) or self.app_label(package) or package,
+                "icon": icon_for_package(package, APP_ICON_FALLBACKS, APP_ICON_GENERIC),
+            }
+            for package in packages
+        ]
+        self._installed_apps = (rows, self._clock())
+        _LOGGER.debug("installed-app discovery found %d apps", len(rows))
+        return rows
+
+    async def _sweep_labels(self, packages: list[str]) -> dict[str, str]:
+        """Labels for a whole sweep, in a bounded burst (t57).
+
+        Deliberately NOT through ``_app_label_requests``: that queue is drained
+        ONE package per watchdog cycle, which is right for the live entity path
+        (a render must never fan out into shells) but would take a couple of
+        minutes for a Shield with ~40 apps. Here the fetches run
+        ``APP_SWEEP_CONCURRENCY`` at a time with a short spacing, so a typical
+        Shield finishes in seconds while the socket still sees a trickle rather
+        than a flood.
+
+        Cached labels are reused, so a second sweep costs almost nothing; a
+        failed fetch is NOT cached (t57 UNIT 0) and the app falls back to its
+        package id for this table while the next sweep retries it.
+        """
+        cached: dict[str, str] = {}
+        missing: list[str] = []
+        for package in packages:
+            label = self.app_label(package)
+            if label:
+                cached[package] = label
+            else:
+                missing.append(package)
+        if not missing:
+            return cached
+
+        semaphore = asyncio.Semaphore(APP_SWEEP_CONCURRENCY)
+
+        async def _one(package: str) -> tuple[str, str | None]:
+            async with semaphore:
+                # a small spacing keeps the burst polite on a single socket
+                await asyncio.sleep(APP_SWEEP_SPACING)
+                return package, await self._dump_label(package)
+
+        results = await asyncio.gather(
+            *(_one(package) for package in missing), return_exceptions=True
+        )
+        for result in results:
+            if isinstance(result, BaseException):
+                continue  # one bad package never fails the sweep
+            package, label = result
+            if label:
+                cached[package] = label
+            else:
+                # None here means the fetch FAILED (nothing cached); the empty
+                # string means the device answered with no label. Both leave
+                # the row on its package id, and both keep the app in the list.
+                cached[package] = ""
+        return cached
+
+
+        """Start at most one label fetch per watchdog cycle."""
+        if not self._app_label_requests or self._poller is None:
+            return
+        package = min(self._app_label_requests)
+        self._app_label_requests.discard(package)
+        self._tasks.append(
+            self._create_task(self._fetch_app_label(package), f"app_label_{package}")
+        )
+
     def _drain_app_label_requests(self) -> None:
         """Start at most one label fetch per watchdog cycle."""
         if not self._app_label_requests or self._poller is None:

custom_components/nvidia_shield/const.py

@@ -151,6 +151,45 @@ DEFAULT_APP_DIRECTORY = {
     "com.google.android.youtube.tv": "YouTube",
 }
 
+# t57: best-effort icon per discovered app. `pm dump` does NOT expose the
+# app's icon resource (an extracted APK's icon is a bitmap only its own code
+# can resolve, and `dumpsys package` prints no drawable reference), so a device
+# icon is not obtainable over plain shell. These MDI names are therefore a
+# HEURISTIC: a curated package -> MDI mapping with a generic fallback, honest
+# about being a guess. The user's own `app_icon_map` (a picture URL or camera
+# entity, see CONF_APP_ICON_MAP) always outranks it for entity_picture.
+APP_ICON_FALLBACKS = {
+    "ar.tvplayer.tv": "mdi:television-classic",
+    "com.github.damontecres.wholphin": "mdi:jellyfish",
+    "com.netflix.ninja": "mdi:netflix",
+    "com.google.android.youtube.tv": "mdi:youtube-tv",
+    "com.google.android.tvlauncher": "mdi:view-dashboard",
+    "org.xbmc.kodi": "mdi:kodi",
+    "com.plexapp.android": "mdi:plex",
+    "com.amazon.amazonvideo.livingroom": "mdi:amazon",
+    "com.disney.disneyplus": "mdi:movie-open",
+    "com.spotify.tv.android": "mdi:spotify",
+    "com.google.android.apps.youtube.music": "mdi:music",
+    "com.valvesoftware.steamlink": "mdi:gamepad-variant",
+    "com.spotify.music": "mdi:spotify",
+}
+#: The fallback for an app this map has never heard of.
+APP_ICON_GENERIC = "mdi:television"
+
+# t57: installed-app discovery. The package list changes only when the user
+# installs or uninstalls something, so a long TTL is right; refreshing is
+# explicit (the options page and the app-list sensor ask for it) rather than a
+# per-poll cost.
+APP_LIST_TTL = 6 * 3600
+#: `pm dump` sweeps for labels run in a small burst — enough to finish a
+#: typical Shield quickly, few enough that the label fetches never queue
+#: behind each other on one socket or stall the watchdog.
+APP_SWEEP_CONCURRENCY = 4
+#: Spacing between the fetches of one sweep, per worker. Keeps the burst a
+#: trickle on the one socket the watchdog uses, while ~40 apps still finish in
+#: a few seconds instead of a few minutes.
+APP_SWEEP_SPACING = 0.05
+
 CHANNEL_DIGIT_DELAY = 0.1
 CONF_CHANNEL_DIGIT_DELAY = "channel_digit_delay"
 CONF_CHANNEL_SEND_ENTER = "channel_send_enter"

custom_components/nvidia_shield/ha_adb_common/__init__.py

@@ -11,6 +11,13 @@ from .const import (
     ATTR_MEDIA_DAG_RUNNING_APPS,
 )
 from .foreground import parse_foreground
+from .installed_apps import (
+    icon_for_package,
+    installed_packages,
+    is_launchable,
+    parse_active_launcher,
+    parse_package_list,
+)
 from .intent import PlaybackIntent
 from .keys import ensure_adb_key
 from .logcat_events import foreground_from_logcat
@@ -38,6 +45,11 @@ __all__ = [
     "parse_audio_playback",
     "parse_foreground",
     "foreground_from_logcat",
+    "icon_for_package",
+    "installed_packages",
+    "is_launchable",
+    "parse_active_launcher",
+    "parse_package_list",
     "parse_wakefulness",
     "parse_media_session",
     "running_apps_from_sessions",

custom_components/nvidia_shield/ha_adb_common/installed_apps.py

@@ -0,0 +1,125 @@
+"""Discover the apps actually installed on the Shield (t57).
+
+The live complaint ("it doesn't find all the apps") came from the curated
+fallback list: only four packages were ever offered, so a user with Wholphin,
+SmartTube, Kodi, Plex or anything else saw nothing. The device knows its own
+apps, so this module asks it.
+
+The commands (both plain `pm`, available on Shield Experience 9 / Android 11
+with no extra permission beyond the ADB pairing this integration already has):
+
+* ``pm list packages -3``      — third-party (user-installed) packages, one
+  ``package:<id>`` line each. This is what the live complaint is about: an app
+  the user installed is a ``-3`` package.
+* ``cmd package resolve-activity --brief -c android.intent.category.HOME``
+  — the ACTIVE launcher (``com.google.android.tvlauncher/....MainActivity``).
+  A launcher is a system app, so it is not in ``-3``, yet it is exactly the app
+  a user sees and may want to name or launch — so it is added explicitly.
+
+System apps are otherwise NOT enumerated: the Shield ships a few hundred of
+them and they are not launchable apps a media-player source list wants. The
+one system app that matters (the launcher) is resolved, not filtered.
+
+Labels come from the existing ``pm dump`` + ``parse_application_label`` path and
+its 24 h cache (``ShieldConnectionManager.app_label``), so discovery adds no
+second naming mechanism.
+
+Icons are best effort and explicitly heuristic — see ``icon_for_package``.
+"""
+
+from __future__ import annotations
+
+import re
+
+#: One `pm list packages` line: ``package:com.example.app``.
+_PACKAGE_LINE_RE = re.compile(r"^package:(?P<package>[\w.]+)\s*$", re.MULTILINE)
+
+#: `resolve-activity --brief` prints ``package/activity`` on the last line,
+#: possibly preceded by the (verbose) intent header.
+_COMPONENT_RE = re.compile(r"^(?P<package>[\w.]+)/[\w.$]+$", re.MULTILINE)
+
+#: Activity components that belong to a launcher but are not what the user
+#: opens: settings/hidden entry points a HOME resolution can land on.
+_LAUNCHER_NOISE = ("com.android.tv.settings",)
+
+
+def parse_package_list(output: str | None) -> list[str]:
+    """The package ids in a ``pm list packages`` dump, in device order.
+
+    Tolerant on purpose: the real output has one ``package:<id>`` per line, and
+    a shell that prepends noise (adb banners, warnings) must not lose the list.
+    Duplicates are collapsed, order preserved.
+    """
+    if not output:
+        return []
+    seen: set[str] = set()
+    packages: list[str] = []
+    for match in _PACKAGE_LINE_RE.finditer(output):
+        package = match.group("package")
+        if package not in seen:
+            seen.add(package)
+            packages.append(package)
+    return packages
+
+
+def parse_active_launcher(output: str | None) -> str | None:
+    """The active HOME package from ``cmd package resolve-activity --brief``.
+
+    Android prints a verbose intent header and then a single
+    ``package/activity`` line; only the last component-looking line is the
+    answer. Returns None when nothing resolved (a device with no launcher, or a
+    command that produced no output) — the caller treats that as "no launcher
+    to add" rather than as a failure.
+    """
+    if not output:
+        return None
+    packages = _COMPONENT_RE.findall(output)
+    for package in reversed(packages):
+        if package not in _LAUNCHER_NOISE:
+            return package
+    return None
+
+
+def installed_packages(user_output: str | None, launcher_output: str | None) -> list[str]:
+    """The discovered package ids: user apps first, then the active launcher.
+
+    The launcher goes last so a source list built from this keeps the user's own
+    apps at the top, and it is de-duplicated because a launcher CAN be a
+    third-party package (a replacement TV launcher installed from the store).
+    """
+    packages = parse_package_list(user_output)
+    launcher = parse_active_launcher(launcher_output)
+    if launcher and launcher not in packages:
+        packages.append(launcher)
+    return packages
+
+
+def icon_for_package(package: str, curated: dict[str, str] | None = None,
+                     generic: str = "mdi:television") -> str:
+    """A best-effort MDI icon for a package, never a non-string (t57).
+
+    HEURISTIC, and labelled as such: the icon is looked up in a curated
+    package -> MDI map, and anything unknown gets the generic fallback. There is
+    no device query behind this because there cannot be one — ``pm dump`` does
+    not print the app's icon resource, and resolving an extracted APK's bitmap
+    would need code the app itself runs. The user's own icon column (a picture
+    URL or a ``camera.*`` entity) is what ``entity_picture`` consumes, and it
+    outranks this.
+
+    Returns a string in every case: the UI is never handed None or a dict.
+    """
+    if not package:
+        return generic
+    table = curated if curated is not None else {}
+    return table.get(package) or generic
+
+
+def is_launchable(package: str) -> bool:
+    """Cheap sanity filter for a package id.
+
+    Not a real Android query: a package id is dotted and non-empty. The filter
+    exists so a malformed line cannot become a table row or a source entry.
+    """
+    return bool(package) and "." in package and all(
+        part.isalnum() or part in ("_", "-") for part in package.split(".")
+    )