chore: log page layout hints to debug Astro redesign issues (#1330)

## ℹ️ Description
Adds debugging output that shows whether and how far a user is affected
by the redesigned (Astro) Kleinanzeigen pages.

- Related issue: #1329 (category selection fails with `No HTML element
found with ID 'cat_185'`); no behavior change, diagnostics only.
- Motivation: reporters currently only see the timeout message.
Diagnostic HTML dumps may contain PII, so users hesitate to share them.

## 📋 Changes Summary

- New `WebScrapingMixin.page_layout_summary()` returns URL,
`astro-island` count, logged-in marker presence, offered `cat_*` IDs,
category picker presence, viewport width and `readyState`. It never
raises.
- `set_category`: DEBUG log of the layout before selection; WARNING with
the layout when a `cat_<id>` element is missing.
- `has_logged_in_marker`: DEBUG log of the layout when no marker is
found.
- `_extract_island_props`: DEBUG log when no Astro island with ad data
exists.
- Publish diagnostics JSON (`capture_on.publish`) now contains
`page_layout`, so no HTML dump is needed to tell the layouts apart.
- German translation for the new warning, docs note in
`CONFIGURATION.md`, unit test for the helper.

Checked the probe against a live logged-in session: start page has 31
islands and the marker, the classic category page has 0 islands and
lists `cat_185`.

### ⚙️ Type of Change
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ] ✨ New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)

Diagnostics/logging only, none of the above applies.

## ✅ Checklist
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
utest`; `pdm run test` additionally launches real browsers and was not
run locally).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Improvements**
* Failed-publish diagnostics now include a page layout summary, helping
distinguish redesigned pages from classic ones without including the
full HTML.
* Additional layout details are recorded when login checks or category
selection encounter problems.
  * Added clearer German translations for category selection errors.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

.github/ISSUE_TEMPLATE/bug.yml

@@ -76,7 +76,10 @@ body:
     id: logs
     attributes:
       label: 📃 Relevant log output (if applicable)
-      description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
+      description: |
+        Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
+
+        For publish problems, also set `diagnostics.capture_on.publish: true` in your `config.yaml`, rerun, and attach the generated `publish_error_*.json` from the diagnostics folder. ⚠️ It contains your full ad config (title, description, contact name/address/phone), so review and redact it before posting. Do not attach the HTML dump or log copy without checking them for personal data (e.g. your account email). See [diagnostics](https://github.com/Second-Hand-Friends/kleinanzeigen-bot/blob/main/docs/CONFIGURATION.md#diagnostics).
     validations:
       required: false
 

docs/CONFIGURATION.md

@@ -423,7 +423,7 @@ The bot uses a layered DOM-first approach to detect login status:
 **Optional diagnostics:**
 
 - Enable `capture_on.login_detection` to capture screenshots and HTML dumps when login detection is inconclusive (`SELECTOR_TIMEOUT`, meaning expected selectors did not appear before timeout)
-- Enable `capture_on.publish` to capture screenshots, HTML dumps, and JSON payloads for each failed publish attempt (e.g., attempts 1–3).
+- Enable `capture_on.publish` to capture screenshots, HTML dumps, and JSON payloads for each failed publish attempt (e.g., attempts 1–3). The JSON contains a `page_layout` summary (URL, number of `astro-island` elements, offered `cat_*` IDs, viewport width) that tells redesigned (Astro) pages from classic ones so the HTML dump is not needed for that. The JSON also holds the ad config (including contact data), so review it before sharing.
 - Enable `capture_log_copy` to copy the entire bot log file when a diagnostic event triggers (e.g., `capture_on.publish` or `capture_on.login_detection`):
   - If multiple diagnostics trigger in the same run, the log will be copied multiple times
   - Review or redact artifacts before sharing publicly

src/kleinanzeigen_bot/app.py

@@ -481,6 +481,7 @@ class KleinanzeigenBot(WebScrapingMixin):  # noqa: PLR0904
             "timestamp": _misc.now().isoformat(timespec = "seconds"),
             "attempt": attempt,
             "page_url": getattr(page, "url", None),
+            "page_layout": await self.page_layout_summary(),
             "exception": {
                 "type": exc.__class__.__name__,
                 "message": str(exc),

src/kleinanzeigen_bot/extract.py

@@ -773,6 +773,7 @@ class AdExtractor(WebScrapingMixin):
                 return null;
             })()""")
             if not raw or not isinstance(raw, str):
+                LOG.debug("No Astro island with ad data found on %s", self.page.url)
                 return {}
             # getAttribute() returns an HTML-decoded value. Parse it directly
             # first so entity-like content inside JSON strings is preserved.

src/kleinanzeigen_bot/login_flow.py

@@ -924,6 +924,7 @@ async def has_logged_in_marker(web:WebScrapingMixin, *, username:str) -> bool:
         else:
             raise
 
+    LOG.debug("Login marker not found. Page layout: %s", await web.page_layout_summary())
     return False
 
 

src/kleinanzeigen_bot/publishing_form.py

@@ -103,6 +103,7 @@ async def set_category(web:WebScrapingMixin, *, category:str | None, ad_file:str
 
     # click on something to trigger automatic category detection
     await web.web_click(By.ID, "ad-description")
+    LOG.debug("Category form layout before selection: %s", await web.page_layout_summary())
 
     is_category_auto_selected = False
     category_path_elem = await web.web_probe(By.ID, "ad-category-path")
@@ -129,6 +130,7 @@ async def set_category(web:WebScrapingMixin, *, category:str | None, ad_file:str
             try:
                 await web.web_click(By.ID, f"cat_{segment}")
             except TimeoutError:
+                LOG.warning("Category element 'cat_%s' not found. Page layout: %s", segment, await web.page_layout_summary())
                 if await web.web_probe(By.ID, "ad-category-picker", timeout = web.timeout("quick_dom")) is None:
                     raise
                 await resolve_category_suggestions(web, category)

src/kleinanzeigen_bot/resources/translations.de.yaml

@@ -275,6 +275,10 @@ kleinanzeigen_bot/published_ads.py:
 #################################################
 kleinanzeigen_bot/publishing_form.py:
 #################################################
+  set_category:
+    "Unknown category alias '%s'. Use a category name from categories.yaml or a numeric category path.": "Unbekannter Kategorie-Alias '%s'. Verwenden Sie einen Kategorienamen aus categories.yaml oder einen numerischen Kategoriepfad."
+    "Category element 'cat_%s' not found. Page layout: %s": "Kategorie-Element 'cat_%s' nicht gefunden. Seitenlayout: %s"
+
   _open_shipping_size_selection:
     "Failed to configure shipping options in dialog!": "Versandoptionen konnten im Dialog nicht konfiguriert werden!"
 
@@ -298,9 +302,6 @@ kleinanzeigen_bot/publishing_form.py:
     "Zip-derived city ('%s') did not converge to expected location: %s": "Aus der PLZ abgeleitete Stadt ('%s') hat sich nicht auf den erwarteten Ort eingestellt: %s"
     "ad-city is a <input readonly>, derived value converged to target '%s'.": "ad-city ist ein <input readonly>, abgeleiteter Wert entspricht dem Zielort '%s'."
 
-  set_category:
-    "Unknown category alias '%s'. Use a category name from categories.yaml or a numeric category path.": "Unbekannter Kategorie-Alias '%s'. Verwenden Sie einen Kategorienamen aus categories.yaml oder einen numerischen Kategoriepfad."
-
   resolve_category_suggestions:
     ? "Category suggestion picker shown, but no segment of configured path '%(category)s' matched the offered suggestions [%(offered)s]. Update the ad's 'category' to an offered ID or a valid full path."
     : "Kategorievorschlag-Auswahl wurde angezeigt, aber kein Segment des konfigurierten Pfades '%(category)s' passte zu den angebotenen Vorschlägen [%(offered)s]. Aktualisieren Sie 'category' auf eine angebotene ID oder einen gültigen vollständigen Pfad."

src/kleinanzeigen_bot/utils/web_scraping_mixin.py

@@ -1369,6 +1369,25 @@ class WebScrapingMixin:  # noqa: PLR0904
 
         return result
 
+    async def page_layout_summary(self) -> dict[str, Any]:
+        """Return layout hints of the current page for troubleshooting; never raises.
+
+        ``astro_islands > 0`` identifies the redesigned (Astro) layout, ``category_ids`` lists the
+        category links currently offered, ``inner_width`` reveals hidden ``md:block`` headers.
+        """
+        try:
+            result = await self.web_execute("""(() => ({
+                astro_islands: document.querySelectorAll('astro-island').length,
+                logged_in_marker: !!document.querySelector('[data-testid="logged-in-user"]'),
+                category_ids: [...document.querySelectorAll('[id^="cat_"]')].map(e => e.id).slice(0, 30),
+                category_picker: !!document.getElementById('ad-category-picker'),
+                inner_width: window.innerWidth,
+                ready_state: document.readyState,
+            }))()""")
+        except Exception as ex:  # noqa: BLE001 diagnostics must never break the workflow
+            return {"unavailable": str(ex)}
+        return {"url": self.page.url, **result} if isinstance(result, dict) else {"url": self.page.url, "raw": result}
+
     async def web_set_input_value(self, element_id:str, value:str) -> None:
         """Sets a framework-controlled input value using the native DOM setter to trigger onChange."""
         await self.web_find(By.ID, element_id)  # raises TimeoutError if element is absent

tests/unit/test_web_scraping_mixin.py

@@ -1076,6 +1076,14 @@ class TestSelectorTimeoutMessages:
 class TestWebScrapingSessionManagement:
     """Test session management edge cases in WebScrapingMixin."""
 
+    @pytest.mark.asyncio
+    async def test_page_layout_summary_reports_layout_and_never_raises(self, web_scraper:WebScrapingMixin) -> None:
+        cast(Any, web_scraper).web_execute = AsyncMock(return_value = {"astro_islands": 4})
+        assert await web_scraper.page_layout_summary() == {"url": "https://example.com", "astro_islands": 4}
+
+        cast(Any, web_scraper).web_execute = AsyncMock(side_effect = RuntimeError("boom"))
+        assert await web_scraper.page_layout_summary() == {"unavailable": "boom"}
+
     @pytest.mark.asyncio
     async def test_close_browser_session_cleans_up_resources(self) -> None:
         """Ensure browser and page references are cleared and child processes are killed."""