Mirrors/kleinanzeigen-bot
1
0
Fork 0
mirror of https://github.com/Second-Hand-Friends/kleinanzeigen-bot.git synced 2026-07-09 12:41:05 +02:00
Code Issues Packages Projects Releases Wiki Activity

refactor: extract delete, extend, download workflows and published-ad fetch into focused modules (#1101)

Browse Source 
## ℹ️ Description

> **Why one PR?** The four extractions are mechanically identical:
extract a method body into a module function, replace `self.*` with
explicit parameters, update call sites in `run()`. Splitting into 4
sequential PRs would produce repetitive diffs against the same
`__init__.py`, compound rebase friction, and add no independent review
signal — each PR would be reviewed as "same pattern, different
function." Combined, the mechanical nature is visible at a glance and
the 17 files are mostly new modules + focused test files.

Reduce KleinanzeigenBot.__init__.py from 2865 to 2391 lines by
extracting four browser workflow modules with explicit dependencies.

## 📋 Changes Summary

- New `published_ads.py` — `fetch_published_ads()` and
`PublishedAdsFetchIncompleteError`, shared by all workflows
- New `delete_flow.py` — `delete_ads()` orchestrator and `delete_ad()`
single-ad deletion
- New `extend_flow.py` — `extend_ads()` orchestrator and `_extend_ad()`
single-ad extension
- New `download_flow.py` — `download_ads()`, `resolve_download_dir()`,
and `_download_ad_with_resolved_state()`
- Promote `_timeout` and `_navigate_paginated_ad_overview` to public in
WebScrapingMixin
- Restore `PublishedAdsFetchIncompleteError` base class to
KleinanzeigenBotError
- New focused test files: `test_published_ads.py` (17),
`test_delete_flow.py` (13), `test_extend_flow.py` (17),
`test_download_flow.py` (16)
- Removed: `test_json_pagination.py`, `test_extend_command.py`
- Flow modules receive explicit dependencies (`web`, `root_url`,
`config`, `workspace`, `load_ads_func`) — no `do_work(bot)` signatures
- Temporary `_fetch_published_ads` delegator kept on bot (still used by
publish/update)

### ⚙️ Type of Change
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [x] ✨ New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)


## ✅ Checklist
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Clarified timeout configuration guidance and examples (use public
timeout API and when to use effective timeout).

* **New Features**
* Added workflows to delete, download (all/new/IDs) and extend ads;
added API-backed published-ads retrieval and safe ID matching.

* **Refactor**
* Command routing moved to workflow modules; unified timeout and
pagination helpers for consistent UI/navigation behavior.

* **Tests**
* Expanded/updated unit tests covering deletion, download, extension,
pagination, timeout and related edge cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Jens
2026-06-12 09:58:56 +02:00
committed by GitHub
parent eeafd64280
commit 7803872422
18 changed files with 2037 additions and 1465 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
CONTRIBUTING.md
View File

@@ -214,10 +214,10 @@ All Python files must start with SPDX license headers:
#### Timeout configuration
- The default timeout (`timeouts.default`) already wraps all standard DOM helpers (`web_find`, `web_click`, etc.) via `WebScrapingMixin._timeout/_effective_timeout`. Use it unless a workflow clearly needs a different SLA.
- Reserve `timeouts.quick_dom` for transient overlays (shipping dialogs, payment prompts, toast banners) that should render almost instantly; call `self._timeout("quick_dom")` in those spots to keep the UI responsive.
- For single selectors that occasionally need more headroom, pass an inline override instead of creating a new config key, e.g. `custom = self._timeout(override = 12.5); await self.web_find(..., timeout = custom)`.
- Use `_timeout()` when you just need the raw configured value (with optional override); use `_effective_timeout()` when you rely on the global multiplier and retry backoff for a given attempt (e.g. inside `_run_with_timeout_retries`).
- The default timeout (`timeouts.default`) already wraps all standard DOM helpers (`web_find`, `web_click`, etc.) via `WebScrapingMixin.timeout/_effective_timeout`. Use it unless a workflow clearly needs a different SLA.
- Reserve `timeouts.quick_dom` for transient overlays (shipping dialogs, payment prompts, toast banners) that should render almost instantly; call `self.timeout("quick_dom")` in those spots to keep the UI responsive.
- For single selectors that occasionally need more headroom, pass an inline override instead of creating a new config key, e.g. `custom = self.timeout(override=12.5); await self.web_find(..., timeout=custom)`.
- Use `timeout()` when you just need the raw configured value (with optional override); use `_effective_timeout()` when you rely on the global multiplier and retry backoff for a given attempt (e.g. inside `_run_with_timeout_retries`).
- Add a new timeout key only when a recurring workflow has its own timing profile (pagination, captcha detection, publishing confirmations, Chrome probes, etc.). Whenever you add one, extend `TimeoutConfig`, document it in the sample `timeouts:` block in `docs/CONFIGURATION.md`, and explain it in `docs/BROWSER_TROUBLESHOOTING.md`.
- Encourage users to raise `timeouts.multiplier` when everything is slow, and override existing keys in `config.yaml` before introducing new ones. This keeps the configuration surface minimal.

628
src/kleinanzeigen_bot/__init__.py
View File

@@ -4,7 +4,6 @@
import asyncio, enum, importlib, json, os, re, sys # isort: skip
import urllib.parse as urllib_parse
from dataclasses import dataclass, replace
from datetime import datetime
from gettext import gettext as _
from pathlib import Path
from typing import TYPE_CHECKING, Any, Final, Sequence, cast
@@ -14,9 +13,8 @@ from nodriver.core.connection import ProtocolException
from ruamel.yaml import YAML
from . import ad_form_helpers as _ad_form_helpers
from . import ad_loading, extract
from . import ad_loading, delete_flow, download_flow, extend_flow, published_ads
from . import ad_state as _ad_state
from . import download_selection as _download_selection
from . import local_path_renaming as _local_path_renaming
from . import price_reduction as _price_reduction
from . import runtime_config as _runtime_config
@@ -33,14 +31,15 @@ from .model.ad_model import (
from .model.ad_model import (
AdUpdateStrategy as AdUpdateStrategy,
)
from .model.config_model import DEFAULT_DOWNLOAD_DIR, Config
from .model.config_model import Config # noqa: TC001 — used at runtime, config injection
from .published_ads import PublishedAd, ad_matches_id
from .update_checker import UpdateChecker
from .utils import diagnostics as _diagnostics
from .utils import dicts as _dicts
from .utils import loggers as _loggers
from .utils import misc as _misc
from .utils import xdg_paths as _xdg_paths
from .utils.exceptions import CaptchaEncountered, CategoryResolutionError, PublishedAdsFetchIncompleteError, PublishSubmissionUncertainError
from .utils.exceptions import CaptchaEncountered, CategoryResolutionError, PublishSubmissionUncertainError
from .utils.files import abspath
from .utils.i18n import pluralize
from .utils.misc import ainput, ensure, is_frozen
@@ -150,44 +149,6 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
def _update_check_state_path(self) -> Path:
return self._workspace_or_raise().state_dir / "update_check_state.json"
def _resolve_download_dir(self) -> Path:
workspace = self._workspace_or_raise()
trimmed_dir = self.config.download.dir.strip()
if trimmed_dir == DEFAULT_DOWNLOAD_DIR:
return workspace.download_dir
return Path(abspath(trimmed_dir, relative_to = str(Path(self.config_file_path).parent))).resolve()
async def _download_ad_with_resolved_state(self, ad_extractor:extract.AdExtractor, ad_id:int, published_ads_by_id:dict[int, dict[str, Any]]) -> None:
"""Download an ad with proper active state resolution and logging.
Resolves the ad's activity state from the published profile, logs appropriately
based on the resolution result, and initiates the download with the resolved state.
This method centralizes the resolution + logging + download logic used by
the "all" and "new" selectors.
Args:
ad_extractor: The AdExtractor instance to use for downloading.
ad_id: The ad ID to download.
published_ads_by_id: Dict mapping ad IDs to published ad data from API.
Note:
The numeric selector does NOT use this helper because it has different
warning message semantics (foreign ads are expected, not anomalies).
"""
resolved = _download_selection.resolve_download_ad_activity(ad_id, published_ads_by_id)
if not resolved.owned:
# Ad not in user's published profile - unexpected for "all"/"new" selectors
# since these only list the user's own ads from the overview page
LOG.warning("Ad %d found in overview but not in published profile. Saving as inactive.", ad_id)
elif not resolved.active:
# Ad is in published profile but not in active state (paused, inactive, etc.)
published_ad = published_ads_by_id.get(ad_id, {})
LOG.debug("Ad %d has state '%s'. Saving as inactive.", ad_id, published_ad.get("state", "unknown"))
await ad_extractor.download_ad(ad_id, active = resolved.active)
async def run(self, args:list[str]) -> None: # noqa: PLR0915
_cli = importlib.import_module(".cli", __name__)
parsed = _cli.parse_args(args)
@@ -342,7 +303,12 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
if ads := self.load_ads():
await self.create_browser_session()
await self.login()
await self.delete_ads(ads)
await delete_flow.delete_ads(
web = self, root_url = self.root_url,
after_delete = self.config.deleting.after_delete,
delete_old_ads_by_title = self.config.publishing.delete_old_ads_by_title,
ad_cfgs = ads,
)
else:
LOG.info("############################################")
LOG.info("DONE: No ads to delete found.")
@@ -364,7 +330,10 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
if ads := self.load_ads():
await self.create_browser_session()
await self.login()
await self.extend_ads(ads)
await extend_flow.extend_ads(
web = self, root_url = self.root_url,
ad_cfgs = ads,
)
else:
LOG.info("############################################")
LOG.info("DONE: No ads found to extend.")
@@ -384,7 +353,14 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
checker.check_for_updates()
await self.create_browser_session()
await self.login()
await self.download_ads()
await download_flow.download_ads(
web = self, config = self.config,
config_file_path = self.config_file_path,
workspace = self._workspace_or_raise(),
ads_selector = self.ads_selector,
load_ads_func = self.load_ads,
root_url = self.root_url,
)
case _:
LOG.error("Unknown command: %s", self.command)
@@ -421,7 +397,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
captcha_elem = await self.web_probe(
By.CSS_SELECTOR,
"iframe[name^='a-'][src^='https://www.google.com/recaptcha/api2/anchor?']",
timeout = self._timeout("captcha_detection"),
timeout = self.timeout("captcha_detection"),
)
context_label = page_context or ("login page" if is_login_page else "publish operation")
@@ -444,8 +420,8 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
async def login(self) -> None:
self._login_detection_diagnostics_captured = False
sso_navigation_timeout = self._timeout("page_load")
pre_login_gdpr_timeout = self._timeout("quick_dom")
sso_navigation_timeout = self.timeout("page_load")
pre_login_gdpr_timeout = self.timeout("quick_dom")
LOG.info("Checking if already logged in...")
await self.web_open(f"{self.root_url}")
@@ -512,7 +488,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
return sanitized or "unknown"
async def _wait_for_auth0_login_context(self) -> None:
redirect_timeout = self._timeout("login_detection")
redirect_timeout = self.timeout("login_detection")
try:
await self.web_await(
lambda: "login.kleinanzeigen.de" in self._current_page_url() or "/u/login" in self._current_page_url(),
@@ -525,7 +501,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
raise AssertionError(_("Auth0 redirect not detected (url=%s)") % current_url) from ex
async def _wait_for_auth0_password_step(self) -> None:
password_step_timeout = self._timeout("login_detection")
password_step_timeout = self.timeout("login_detection")
try:
await self.web_await(
lambda: "/u/login/password" in self._current_page_url(),
@@ -538,8 +514,8 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
raise AssertionError(_("Auth0 password step not reached (url=%s)") % current_url) from ex
async def _wait_for_post_auth0_submit_transition(self) -> None:
post_submit_timeout = self._timeout("login_detection")
quick_dom_timeout = self._timeout("quick_dom")
post_submit_timeout = self.timeout("login_detection")
quick_dom_timeout = self.timeout("quick_dom")
fallback_max_ms = max(700, int(quick_dom_timeout * 1_000))
fallback_min_ms = max(300, fallback_max_ms // 2)
@@ -625,7 +601,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
await self._click_gdpr_banner()
async def _check_sms_verification(self) -> None:
sms_timeout = self._timeout("sms_verification")
sms_timeout = self.timeout("sms_verification")
element = await self.web_probe(By.TEXT, "Wir haben dir gerade einen 6-stelligen Code für die Telefonnummer", timeout = sms_timeout)
if element is None:
LOG.debug("No SMS verification prompt detected after login")
@@ -642,7 +618,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
all form interaction until dismissed. Uses a short timeout to avoid slowing down
the flow when the banner is already gone.
"""
banner_timeout = self._timeout("quick_dom")
banner_timeout = self.timeout("quick_dom")
element = await self.web_probe(By.ID, "gdpr-banner-accept", timeout = banner_timeout)
if element is not None:
LOG.debug("Consent banner detected, clicking 'Alle akzeptieren'...")
@@ -652,7 +628,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
LOG.debug("Consent banner not present; continuing without dismissal")
async def _check_email_verification(self) -> None:
email_timeout = self._timeout("email_verification")
email_timeout = self.timeout("email_verification")
element = await self.web_probe(By.TEXT, "Um dein Konto zu schützen haben wir dir eine E-Mail geschickt", timeout = email_timeout)
if element is None:
LOG.debug("No email verification prompt detected after login")
@@ -663,7 +639,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
await ainput(_("Press ENTER when done..."))
async def _click_gdpr_banner(self, *, timeout:float | None = None) -> None:
gdpr_timeout = self._timeout("quick_dom") if timeout is None else timeout
gdpr_timeout = self.timeout("quick_dom") if timeout is None else timeout
element = await self.web_probe(By.ID, "gdpr-banner-accept", timeout = gdpr_timeout)
if element is not None:
await element.click()
@@ -808,7 +784,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
# to allow sufficient time for client-side JavaScript rendering after page load.
# This is especially important for older sessions (20+ days) that require
# additional server-side validation time.
login_check_timeout = self._timeout("login_detection")
login_check_timeout = self.timeout("login_detection")
effective_timeout = self._effective_timeout("login_detection")
username = self.config.login.username.lower()
LOG.debug(
@@ -816,7 +792,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
login_check_timeout,
effective_timeout,
)
quick_dom_timeout = self._timeout("quick_dom")
quick_dom_timeout = self.timeout("quick_dom")
tried_login_selectors = _format_login_detection_selectors(_LOGIN_DETECTION_SELECTORS)
try:
@@ -872,7 +848,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
# If false positives occur, harden by adding web_check(Is.DISPLAYED) on cta_element.
# See issue #876.
async def _has_logged_out_cta(self, *, log_timeout:bool = True) -> bool:
quick_dom_timeout = self._timeout("quick_dom")
quick_dom_timeout = self.timeout("quick_dom")
tried_logged_out_selectors = _format_login_detection_selectors(_LOGGED_OUT_CTA_SELECTORS)
try:
@@ -904,354 +880,9 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
return False
async def _fetch_published_ads(self, *, strict:bool = False) -> list[dict[str, Any]]:
"""Fetch all published ads, handling API pagination.
Args:
strict: If True, raise PublishedAdsFetchIncompleteError when pagination data is incomplete.
Returns:
List of all published ads across all pages.
"""
ads:list[dict[str, Any]] = []
page = 1
MAX_PAGE_LIMIT:Final[int] = 100
SNIPPET_LIMIT:Final[int] = 500
def _handle_incomplete_fetch(template:str, *args:Any, cause:Exception | None = None) -> None:
if strict:
raise PublishedAdsFetchIncompleteError(_(template) % args) from cause
while True:
# Safety check: don't paginate beyond reasonable limit
if page > MAX_PAGE_LIMIT:
LOG.warning("Stopping pagination after %s pages to avoid infinite loop", MAX_PAGE_LIMIT)
_handle_incomplete_fetch("Stopping pagination after %s pages to avoid infinite loop", MAX_PAGE_LIMIT)
break
try:
response = await self.web_request(f"{self.root_url}/m-meine-anzeigen-verwalten.json?sort=DEFAULT&pageNum={page}")
except TimeoutError as ex:
LOG.warning("Pagination request failed on page %s: %s", page, ex)
_handle_incomplete_fetch("Pagination request failed on page %s: %s", page, ex, cause = ex)
break
if not isinstance(response, dict):
LOG.warning("Unexpected pagination response type on page %s: %s", page, type(response).__name__)
_handle_incomplete_fetch("Unexpected pagination response type on page %s: %s", page, type(response).__name__)
break
content = response.get("content", "")
if isinstance(content, bytearray):
content = bytes(content)
if isinstance(content, bytes):
content = content.decode("utf-8", errors = "replace")
if not isinstance(content, str):
LOG.warning("Unexpected response content type on page %s: %s", page, type(content).__name__)
_handle_incomplete_fetch("Unexpected response content type on page %s: %s", page, type(content).__name__)
break
try:
json_data = json.loads(content)
except (json.JSONDecodeError, TypeError) as ex:
if not content:
LOG.warning("Empty JSON response content on page %s", page)
_handle_incomplete_fetch("Empty JSON response content on page %s", page, cause = ex)
break
snippet = content[:SNIPPET_LIMIT] + ("..." if len(content) > SNIPPET_LIMIT else "")
LOG.warning("Failed to parse JSON response on page %s: %s (content: %s)", page, ex, snippet)
_handle_incomplete_fetch("Failed to parse JSON response on page %s: %s (content: %s)", page, ex, snippet, cause = ex)
break
if not isinstance(json_data, dict):
snippet = content[:SNIPPET_LIMIT] + ("..." if len(content) > SNIPPET_LIMIT else "")
LOG.warning("Unexpected JSON payload on page %s (content: %s)", page, snippet)
_handle_incomplete_fetch("Unexpected JSON payload on page %s (content: %s)", page, snippet)
break
page_ads = json_data.get("ads", [])
if not isinstance(page_ads, list):
preview = str(page_ads)
if len(preview) > SNIPPET_LIMIT:
preview = preview[:SNIPPET_LIMIT] + "..."
LOG.warning("Unexpected 'ads' type on page %s: %s value: %s", page, type(page_ads).__name__, preview)
_handle_incomplete_fetch("Unexpected 'ads' type on page %s: %s value: %s", page, type(page_ads).__name__, preview)
break
filtered_page_ads:list[dict[str, Any]] = []
rejected_count = 0
rejected_preview:str | None = None
for entry in page_ads:
if isinstance(entry, dict) and "id" in entry and "state" in entry:
filtered_page_ads.append(entry)
continue
rejected_count += 1
if rejected_preview is None:
rejected_preview = repr(entry)
if rejected_count > 0:
preview = rejected_preview or "<none>"
if len(preview) > SNIPPET_LIMIT:
preview = preview[:SNIPPET_LIMIT] + "..."
LOG.warning("Filtered %s malformed ad entries on page %s (sample: %s)", rejected_count, page, preview)
_handle_incomplete_fetch("Filtered %s malformed ad entries on page %s (sample: %s)", rejected_count, page, preview)
ads.extend(filtered_page_ads)
paging = json_data.get("paging")
if not isinstance(paging, dict):
LOG.debug("No paging dict found on page %s, assuming single page", page)
break
# Use only real API fields (confirmed from production data)
current_page_num = _misc.coerce_page_number(paging.get("pageNum"))
total_pages = _misc.coerce_page_number(paging.get("last"))
if current_page_num is None:
LOG.warning("Invalid 'pageNum' in paging info: %s, stopping pagination", paging.get("pageNum"))
_handle_incomplete_fetch("Invalid 'pageNum' in paging info: %s, stopping pagination", paging.get("pageNum"))
break
# Stop if reached last page (only when API provides 'last')
if total_pages is not None and current_page_num >= total_pages:
LOG.info("Reached last page %s of %s, stopping pagination", current_page_num, total_pages)
break
# Safety: stop if no ads returned
if len(page_ads) == 0:
LOG.info("No ads found on page %s, stopping pagination", page)
break
LOG.debug("Page %s: fetched %s ads (numFound=%s)", page, len(page_ads), paging.get("numFound"))
# Use API's next field for navigation (more robust than our counter)
next_page = _misc.coerce_page_number(paging.get("next"))
if next_page is None:
if total_pages is not None:
LOG.warning("Invalid 'next' page value in paging info: %s, stopping pagination", paging.get("next"))
_handle_incomplete_fetch("Invalid 'next' page value in paging info: %s, stopping pagination", paging.get("next"))
else:
LOG.debug("No 'next' in paging on page %s, assuming last page", page)
_handle_incomplete_fetch("No 'next' in paging on page %s, assuming last page", page)
break
page = next_page
return ads
async def delete_ads(self, ad_cfgs:list[tuple[str, Ad, dict[str, Any]]]) -> None:
count = 0
deleted_count = 0
after_delete = self.config.deleting.after_delete
published_ads = await self._fetch_published_ads()
for ad_file, ad_cfg, ad_cfg_orig in ad_cfgs:
count += 1
LOG.info("Processing %s/%s: '%s' from [%s]...", count, len(ad_cfgs), ad_cfg.title, ad_file)
# Record pre-delete id to detect whether delete_ad attempted a deletion.
# delete_ad clears ad_cfg.id when targets were found (Phase B ran),
# and preserves it on no-match early return.
id_before = ad_cfg.id
deleted = await self.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = self.config.publishing.delete_old_ads_by_title)
if deleted:
deleted_count += 1
# Apply after_delete policy only when a delete was actually attempted.
# Detection: True return (some 200), or id changed from non-None to None (all 404).
# When id was already None before the call, only True return is reliable;
# a False return could be no-match or title-match all-404, both treated as no cleanup.
delete_attempted = deleted or (id_before is not None and ad_cfg.id is None)
if delete_attempted and after_delete != "NONE":
if _ad_state.apply_after_delete_policy(ad_cfg, ad_cfg_orig, mode = after_delete):
_dicts.save_dict(ad_file, ad_cfg_orig)
await self.web_sleep()
LOG.info("############################################")
LOG.info("DONE: Deleted %s of %s", deleted_count, pluralize("ad", count))
LOG.info("############################################")
async def delete_ad(self, ad_cfg:Ad, published_ads:list[dict[str, Any]], *, delete_old_ads_by_title:bool) -> bool:
"""Delete an ad from the server.
Returns:
True if at least one delete request returned 200 (confirmed deleted).
False if no matching ads were found or all returned 404 (already gone).
Side effects:
Clears ``ad_cfg.id`` whenever a delete was attempted, regardless of
the server response (200 or 404). The old ID is stale in both cases.
Preserves ``ad_cfg.id`` when no deletion was attempted (early return).
"""
LOG.info("Deleting ad '%s' if already present...", ad_cfg.title)
# Phase A: Build set of IDs to delete, unified across both modes
ids_to_delete:set[int] = set()
if delete_old_ads_by_title:
for published_ad in published_ads:
raw_id = published_ad.get("id")
if raw_id is None:
LOG.debug("Skipping published ad with missing id: %r", published_ad.get("title"))
continue
try:
published_ad_id = int(raw_id)
except (ValueError, TypeError):
LOG.debug("Skipping published ad with invalid id: %r", raw_id)
continue
published_ad_title = published_ad.get("title", "")
if ad_cfg.id == published_ad_id or ad_cfg.title == published_ad_title:
LOG.debug(" -> matched ad %s '%s' for deletion", published_ad_id, published_ad_title)
ids_to_delete.add(published_ad_id)
elif ad_cfg.id is not None:
ids_to_delete.add(ad_cfg.id)
# Early return if nothing to delete — skip page open, CSRF fetch, and sleep
if not ids_to_delete:
LOG.info(" -> SKIPPED: no published ad matched '%s' for deletion", ad_cfg.title)
return False
# Phase B: Open manage-ads page, fetch CSRF token, execute deletions
await self.web_open(f"{self.root_url}/m-meine-anzeigen.html")
csrf_token_elem = await self.web_find(By.CSS_SELECTOR, "meta[name=_csrf]")
csrf_token = csrf_token_elem.attrs["content"]
ensure(csrf_token is not None, "Expected CSRF Token not found in HTML content!")
HTTP_OK:Final = 200
deleted = False
for target_id in ids_to_delete:
LOG.debug(" -> deleting ad %s...", target_id)
response = await self.web_request(
url = f"{self.root_url}/m-anzeigen-loeschen.json?ids={target_id}",
method = "POST",
headers = {"x-csrf-token": str(csrf_token)},
valid_response_codes = [200, 404],
)
if response["statusCode"] == HTTP_OK:
deleted = True
LOG.info(" -> SUCCESS: deleted ad '%s' (ID: %s)", ad_cfg.title, target_id)
else:
LOG.warning(" -> ad %s not found (status %s), may have been removed already", target_id, response["statusCode"])
await self.web_sleep()
# Clear ad_cfg.id whenever a delete was attempted — the old ID is stale
# regardless of whether the server returned 200 (deleted) or 404 (already gone).
ad_cfg.id = None
return deleted
async def extend_ads(self, ad_cfgs:list[tuple[str, Ad, dict[str, Any]]]) -> None:
"""Extends ads that are close to expiry."""
# Fetch currently published ads from API
published_ads = await self._fetch_published_ads()
# Filter ads that need extension
ads_to_extend = []
for ad_file, ad_cfg, ad_cfg_orig in ad_cfgs:
# Skip unpublished ads (no ID)
if not ad_cfg.id:
LOG.info(" -> SKIPPED: ad '%s' is not published yet", ad_cfg.title)
continue
# Find ad in published list
published_ad = next((ad for ad in published_ads if ad["id"] == ad_cfg.id), None)
if not published_ad:
LOG.warning(" -> SKIPPED: ad '%s' (ID: %s) not found in published ads", ad_cfg.title, ad_cfg.id)
continue
# Skip non-active ads
if published_ad.get("state") != "active":
LOG.info(" -> SKIPPED: ad '%s' is not active (state: %s)", ad_cfg.title, published_ad.get("state"))
continue
# Check if ad is within 8-day extension window using API's endDate
end_date_str = published_ad.get("endDate")
if not end_date_str:
LOG.warning(" -> SKIPPED: ad '%s' has no endDate in API response", ad_cfg.title)
continue
# Intentionally parsing naive datetime from kleinanzeigen API's German date format, timezone not relevant for date-only comparison
end_date = datetime.strptime(end_date_str, "%d.%m.%Y") # noqa: DTZ007
days_until_expiry = (end_date.date() - _misc.now().date()).days
# Magic value 8 is kleinanzeigen.de's platform policy: extensions only possible within 8 days of expiry
if days_until_expiry <= 8: # noqa: PLR2004
LOG.info(" -> ad '%s' expires in %d days, will extend", ad_cfg.title, days_until_expiry)
ads_to_extend.append((ad_file, ad_cfg, ad_cfg_orig, published_ad))
else:
LOG.info(" -> SKIPPED: ad '%s' expires in %d days (can only extend within 8 days)", ad_cfg.title, days_until_expiry)
if not ads_to_extend:
LOG.info("No ads need extension at this time.")
LOG.info("############################################")
LOG.info("DONE: No ads extended.")
LOG.info("############################################")
return
# Process extensions
success_count = 0
for idx, (ad_file, ad_cfg, ad_cfg_orig, _published_ad) in enumerate(ads_to_extend, start = 1):
LOG.info("Processing %s/%s: '%s' from [%s]...", idx, len(ads_to_extend), ad_cfg.title, ad_file)
if await self.extend_ad(ad_file, ad_cfg, ad_cfg_orig):
success_count += 1
await self.web_sleep()
LOG.info("############################################")
LOG.info("DONE: Extended %s", pluralize("ad", success_count))
LOG.info("############################################")
async def extend_ad(self, ad_file:str, ad_cfg:Ad, ad_cfg_orig:dict[str, Any]) -> bool:
"""Extends a single ad listing."""
LOG.info("Extending ad '%s' (ID: %s)...", ad_cfg.title, ad_cfg.id)
try:
# Navigate to ad management page and find extend button across all pages
extend_button_xpath = f'//li[@data-adid="{ad_cfg.id}"]//button[contains(., "Verlängern")]'
async def find_and_click_extend_button(page_num:int) -> bool:
"""Try to find and click extend button on current page."""
try:
extend_button = await self.web_find(By.XPATH, extend_button_xpath, timeout = self._timeout("quick_dom"))
LOG.info("Found extend button on page %s", page_num)
await extend_button.click()
return True # Success - stop pagination
except TimeoutError:
LOG.debug("Extend button not found on page %s", page_num)
return False # Continue to next page
success = await self._navigate_paginated_ad_overview(find_and_click_extend_button, page_url = f"{self.root_url}/m-meine-anzeigen.html")
if not success:
LOG.error(" -> FAILED: Could not find extend button for ad ID %s", ad_cfg.id)
return False
# Handle confirmation dialog
# After clicking "Verlängern", a dialog appears with:
# - Title: "Vielen Dank!"
# - Message: "Deine Anzeige ... wurde erfolgreich verlängert."
# - Paid bump-up option (skipped by closing dialog)
# Simply close the dialog with the X button (aria-label="Schließen")
try:
dialog_close_timeout = self._timeout("quick_dom")
await self.web_click(By.CSS_SELECTOR, 'button[aria-label="Schließen"]', timeout = dialog_close_timeout)
LOG.debug(" -> Closed confirmation dialog")
except TimeoutError:
LOG.warning(" -> No confirmation dialog found, extension may have completed directly")
# Update metadata in YAML file
# Update updated_on to track when ad was extended
ad_cfg_orig["updated_on"] = _misc.now().isoformat(timespec = "seconds")
_dicts.save_dict(ad_file, ad_cfg_orig)
LOG.info(" -> SUCCESS: ad extended with ID %s", ad_cfg.id)
return True
except TimeoutError as ex:
LOG.error(" -> FAILED: Timeout while extending ad '%s': %s", ad_cfg.title, ex)
return False
except OSError as ex:
LOG.error(" -> FAILED: Could not persist extension for ad '%s': %s", ad_cfg.title, ex)
return False
async def _fetch_published_ads(self, *, strict:bool = False) -> list[PublishedAd]:
"""Temporary delegator to published_ads.fetch_published_ads."""
return await published_ads.fetch_published_ads(self, self.root_url, strict = strict)
async def __check_publishing_result(self) -> bool:
# Check for success messages
@@ -1306,7 +937,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
for idx, (ad_file, ad_cfg, ad_cfg_orig) in enumerate(ad_cfgs, start = 1):
LOG.info("Processing %s/%s: '%s' from [%s]...", idx, len(ad_cfgs), ad_cfg.title, ad_file)
if [x for x in published_ads if x["id"] == ad_cfg.id and x["state"] == "paused"]:
if any(ad_matches_id(x, ad_cfg.id) and x.get("state") == "paused" for x in published_ads):
LOG.info("Skipping because ad is reserved")
continue
@@ -1360,13 +991,18 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
# Check publishing result separately (no retry - ad is already submitted)
if success:
try:
publish_timeout = self._timeout("publishing_result")
publish_timeout = self.timeout("publishing_result")
await self.web_await(self.__check_publishing_result, timeout = publish_timeout)
except TimeoutError:
LOG.warning(" -> Could not confirm publishing for '%s', but ad may be online", ad_cfg.title)
if success and self.config.publishing.delete_old_ads == "AFTER_PUBLISH" and not self.keep_old_ads:
await self.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = False)
await delete_flow.delete_ad(
web = self, root_url = self.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = False,
)
LOG.info("############################################")
if failed_count > 0:
@@ -1376,7 +1012,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
LOG.info("############################################")
async def publish_ad( # noqa: PLR0915 PLR0914 PLR0912
self, ad_file:str, ad_cfg:Ad, ad_cfg_orig:dict[str, Any], published_ads:list[dict[str, Any]], mode:AdUpdateStrategy = AdUpdateStrategy.REPLACE
self, ad_file:str, ad_cfg:Ad, ad_cfg_orig:dict[str, Any], published_ads:list[PublishedAd], mode:AdUpdateStrategy = AdUpdateStrategy.REPLACE
) -> None:
"""Publish or update an ad on Kleinanzeigen.
@@ -1396,7 +1032,12 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
if mode == AdUpdateStrategy.REPLACE:
if self.config.publishing.delete_old_ads == "BEFORE_PUBLISH" and not self.keep_old_ads:
await self.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = self.config.publishing.delete_old_ads_by_title)
await delete_flow.delete_ad(
web = self, root_url = self.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = self.config.publishing.delete_old_ads_by_title,
)
# Apply auto price reduction in REPLACE mode (republish flow)
_price_reduction.apply_auto_price_reduction(
@@ -1453,7 +1094,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
shipping_btn = await self.web_find(
By.CSS_SELECTOR,
'[role="combobox"][id$=".versand"]',
timeout = self._timeout("quick_dom"),
timeout = self.timeout("quick_dom"),
)
btn_id = cast(str, shipping_btn.attrs.get("id"))
if not btn_id:
@@ -1488,7 +1129,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
#############################
if ad_cfg.type != "WANTED":
sell_directly = ad_cfg.sell_directly
quick_dom = self._timeout("quick_dom")
quick_dom = self.timeout("quick_dom")
if ad_cfg.shipping_type == "SHIPPING":
if sell_directly and price_type in {"FIXED", "NEGOTIABLE"}:
buy_now_true = await self.web_probe(By.ID, "ad-buy-now-true", timeout = quick_dom)
@@ -1521,7 +1162,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
#############################
remove_button_selector = "button[aria-label='Bild entfernen']"
hidden_marker_selector = "input[name^='adImages'][name$='.url']"
quick_dom = self._timeout("quick_dom")
quick_dom = self.timeout("quick_dom")
removed_count = 0
try:
@@ -1577,7 +1218,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
# PostListingForm v2 may show an "Effektiver verkaufen" upsell
# dialog after clicking submit. Dismiss it so the actual form
# POST can proceed.
quick_dom = self._timeout("quick_dom")
quick_dom = self.timeout("quick_dom")
upsell_dialog = await self.web_probe(
By.XPATH, "//dialog[@open and contains(., 'Effektiver verkaufen')]", timeout = quick_dom
)
@@ -1592,7 +1233,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
# Everything after the first click is uncertain: the ad may already have been submitted.
ad_id:int | None = None
try:
quick_dom = self._timeout("quick_dom")
quick_dom = self.timeout("quick_dom")
imprint_btn = await self.web_probe(By.ID, "imprint-guidance-submit", timeout = quick_dom)
if imprint_btn is not None:
@@ -1616,7 +1257,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
await self.web_scroll_page_down()
await ainput(_("Press a key to continue..."))
confirmation_timeout = self._timeout("publishing_confirmation")
confirmation_timeout = self.timeout("publishing_confirmation")
async def _check_confirmation_url() -> bool:
url = str(await self.web_execute("window.location.href"))
@@ -1822,8 +1463,8 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
return ""
async def __read_city_selection_text(self) -> str | None:
city_timeout = self._timeout("default")
quick_dom_timeout = self._timeout("quick_dom")
city_timeout = self.timeout("default")
quick_dom_timeout = self.timeout("quick_dom")
try:
city_element = await self.web_find(By.ID, "ad-city", timeout = city_timeout)
except TimeoutError:
@@ -1857,8 +1498,8 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
return None
async def __select_city_combobox_option(self, target:str) -> None:
quick_dom_timeout = self._timeout("quick_dom")
city_flow_timeout = self._timeout("default")
quick_dom_timeout = self.timeout("quick_dom")
city_flow_timeout = self.timeout("default")
await self.web_click(By.ID, "ad-city", timeout = quick_dom_timeout)
city_element = await self.web_find(By.ID, "ad-city", timeout = quick_dom_timeout)
@@ -1936,7 +1577,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
if self.__location_matches_target(target, selected_city):
return
city_timeout = self._timeout("default")
city_timeout = self.timeout("default")
city_element = await self.web_find(By.ID, "ad-city", timeout = city_timeout)
if city_element is None:
raise TimeoutError(_("Unsupported city element type while setting contact location: <%s>") % "missing")
@@ -2000,15 +1641,15 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
# set contact phone
#############################
if contact.phone:
phone_elem = await self.web_probe(By.ID, "ad-phone", timeout = self._timeout("quick_dom"))
phone_elem = await self.web_probe(By.ID, "ad-phone", timeout = self.timeout("quick_dom"))
if phone_elem is None:
LOG.info(
"Phone number field not present on page. This is expected for many private accounts; commercial accounts may still support phone numbers."
)
else:
try:
if await self.web_check(By.ID, "ad-phone", Is.DISABLED, timeout = self._timeout("quick_dom")):
await self.web_click(By.ID, "ad-phone-visibility", timeout = self._timeout("quick_dom"))
if await self.web_check(By.ID, "ad-phone", Is.DISABLED, timeout = self.timeout("quick_dom")):
await self.web_click(By.ID, "ad-phone-visibility", timeout = self.timeout("quick_dom"))
await self.web_sleep()
await self.__set_input_value("ad-phone", contact.phone)
except TimeoutError as ex:
@@ -2035,7 +1676,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
for idx, (ad_file, ad_cfg, ad_cfg_orig) in enumerate(ad_cfgs, start = 1):
LOG.info("Processing %s/%s: '%s' from [%s]...", idx, len(ad_cfgs), ad_cfg.title, ad_file)
ad = next((ad for ad in published_ads if ad["id"] == ad_cfg.id), None)
ad = next((published_ad for published_ad in published_ads if ad_matches_id(published_ad, ad_cfg.id)), None)
if not ad:
LOG.warning(" -> SKIPPED: ad '%s' (ID: %s) not found in published ads", ad_cfg.title, ad_cfg.id)
@@ -2088,7 +1729,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
if success:
try:
publish_timeout = self._timeout("publishing_result")
publish_timeout = self.timeout("publishing_result")
await self.web_await(self.__check_publishing_result, timeout = publish_timeout)
except TimeoutError:
LOG.warning(" -> Could not confirm update for '%s', but changes may be online", ad_cfg.title)
@@ -2110,7 +1751,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
if legacy_value is not None:
LOG.warning("Condition value [%s] is deprecated; update your config to [%s].", legacy_value, canonical_value)
short_timeout = self._timeout("quick_dom")
short_timeout = self.timeout("quick_dom")
condition_trigger_xpath = "//label[contains(@for, '.condition')]/following::button[@aria-haspopup='dialog' or @aria-haspopup='true'][1]"
condition_trigger = await self.web_probe(By.XPATH, condition_trigger_xpath, timeout = short_timeout)
@@ -2217,7 +1858,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
of the segments match — surfaces an actionable error instead of letting
the submit retry loop trip the duplicate-guard.
"""
picker_timeout = self._timeout("quick_dom")
picker_timeout = self.timeout("quick_dom")
picker = await self.web_probe(By.ID, "ad-category-picker", timeout = picker_timeout)
if picker is None:
return
@@ -2369,7 +2010,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
f" or contains(@name, {original_key_literal})"
"]"
)
quick_dom = self._timeout("quick_dom")
quick_dom = self.timeout("quick_dom")
try:
if special_attribute_key == "condition_s":
special_attr_probe = await self.web_probe(By.XPATH, special_attr_xpath, timeout = quick_dom)
@@ -2499,7 +2140,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
raise TimeoutError(_("Option '%(value)s' not found in button combobox '%(id)s'") % {"value": value, "id": elem_id})
async def __set_shipping(self, ad_cfg:Ad, mode:AdUpdateStrategy = AdUpdateStrategy.REPLACE) -> None:
short_timeout = self._timeout("quick_dom")
short_timeout = self.timeout("quick_dom")
if ad_cfg.shipping_type == "PICKUP":
pickup_radio = await self.web_probe(By.ID, "ad-shipping-enabled-no", timeout = short_timeout)
if pickup_radio is None:
@@ -2632,7 +2273,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
wanted_codes = set(wanted_carrier_codes)
all_codes_for_size = CARRIER_CODES_BY_SIZE[shipping_size]
short_timeout = self._timeout("quick_dom")
short_timeout = self.timeout("quick_dom")
dialog = '//*[self::dialog or @role="dialog"]'
try:
@@ -2683,7 +2324,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
LOG.info(" -> found %s", pluralize("image", ad_cfg.images))
hidden_marker_selector = "input[name^='adImages'][name$='.url']"
quick_dom_timeout = self._timeout("quick_dom")
quick_dom_timeout = self.timeout("quick_dom")
# Capture marker baseline before this upload attempt to avoid counting stale values
baseline_marker_count = 0
@@ -2723,7 +2364,7 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
return current_count >= expected_count
try:
await self.web_await(check_thumbnails_uploaded, timeout = self._timeout("image_upload"), timeout_error_message = _("Image upload timeout exceeded"))
await self.web_await(check_thumbnails_uploaded, timeout = self.timeout("image_upload"), timeout_error_message = _("Image upload timeout exceeded"))
except TimeoutError as ex:
# Get current count for better error message
current_count = await count_processed_images()
@@ -2734,121 +2375,6 @@ class KleinanzeigenBot(WebScrapingMixin): # noqa: PLR0904
LOG.info(" -> all images uploaded successfully")
async def download_ads(self) -> None:
"""
Determines which download mode was chosen with the arguments, and calls the specified download routine.
This downloads either all, only unsaved(new), or specific ads given by ID.
"""
# Normalize comma-separated keyword selectors; set deduplication collapses "new,new" → {"new"}
selector_tokens = {s.strip() for s in self.ads_selector.split(",")}
if "all" in selector_tokens:
effective_selector = "all"
elif len(selector_tokens) == 1:
effective_selector = next(iter(selector_tokens)) # e.g. "new,new" → "new"
else:
effective_selector = self.ads_selector # numeric IDs: "123,456" — unchanged
# Fetch published ads once from manage-ads JSON to avoid repetitive API calls during extraction
# Build lookup dict inline and pass directly to extractor (no cache abstraction needed)
LOG.info("Fetching ad metadata (status, expiry dates)...")
published_ads = await self._fetch_published_ads(strict = _download_selection.is_numeric_ids_selector(effective_selector))
published_ads_by_id:dict[int, dict[str, Any]] = {}
for published_ad in published_ads:
try:
ad_id = published_ad.get("id")
if ad_id is not None:
published_ads_by_id[int(ad_id)] = published_ad
except (ValueError, TypeError):
LOG.warning("Skipping ad with non-numeric id: %s", published_ad.get("id"))
LOG.info("Loaded metadata for %s published ads.", len(published_ads_by_id))
download_dir = self._resolve_download_dir()
_xdg_paths.ensure_directory(download_dir, "downloaded ads directory")
LOG.info("Ads download directory: %s", download_dir)
ad_extractor = extract.AdExtractor(self.browser, self.config, download_dir, published_ads_by_id = published_ads_by_id)
if effective_selector in {"all", "new"}: # explore ads overview for these two modes
LOG.info("Scanning ad overview for navigation URLs...")
own_ad_urls = await ad_extractor.extract_own_ads_urls()
LOG.info("Found %s.", pluralize("ad URL", len(own_ad_urls)))
if effective_selector == "all": # download all of your ads
LOG.info("Starting download of all ads...")
valid_ad_refs:list[tuple[str, int]] = []
for ad_url in own_ad_urls:
ad_id = ad_extractor.extract_ad_id_from_ad_url(ad_url)
if ad_id == -1:
# Skip ads with invalid URLs (warning already logged by extract_ad_id_from_ad_url)
continue
valid_ad_refs.append((ad_url, ad_id))
success_count = 0
# call download function for each ad page
for idx, (ad_url, ad_id) in enumerate(valid_ad_refs, start = 1):
LOG.info("Downloading %d/%d ads...", idx, len(valid_ad_refs))
if await ad_extractor.navigate_to_ad_page(ad_url):
await self._download_ad_with_resolved_state(ad_extractor, ad_id, published_ads_by_id)
success_count += 1
LOG.info("%d of %d ads were downloaded from your profile.", success_count, len(valid_ad_refs))
elif effective_selector == "new": # download only unsaved ads
# check which ads already saved
saved_ad_ids = []
ads = self.load_ads(ignore_inactive = False, exclude_ads_with_id = False) # do not skip because of existing IDs
for ad in ads:
saved_ad_id = ad[1].id
if saved_ad_id is None:
LOG.debug("Skipping saved ad without id (likely unpublished or manually created): %s", ad[0])
continue
saved_ad_ids.append(int(saved_ad_id))
# determine ad IDs from links
ad_id_by_url = {url: ad_extractor.extract_ad_id_from_ad_url(url) for url in own_ad_urls}
LOG.info("Starting download of not yet downloaded ads...")
ads_to_download:list[tuple[str, int]] = []
for ad_url, ad_id in ad_id_by_url.items():
# Skip ads with invalid URLs (warning already logged by extract_ad_id_from_ad_url)
if ad_id == -1:
continue
# check if ad with ID already saved
if ad_id in saved_ad_ids:
LOG.info("The ad with id %d has already been saved.", ad_id)
continue
ads_to_download.append((ad_url, ad_id))
new_count = 0
for idx, (ad_url, ad_id) in enumerate(ads_to_download, start = 1):
LOG.info("Downloading %d/%d ads...", idx, len(ads_to_download))
if await ad_extractor.navigate_to_ad_page(ad_url):
await self._download_ad_with_resolved_state(ad_extractor, ad_id, published_ads_by_id)
new_count += 1
LOG.info("%s were downloaded from your profile.", pluralize("new ad", new_count))
elif _download_selection.is_numeric_ids_selector(effective_selector): # download ad(s) with specific id(s)
ids = [int(n) for n in effective_selector.split(",")]
LOG.info("Starting download of ad(s) with the id(s):")
LOG.info(" | ".join([str(ad_id) for ad_id in ids]))
for idx, ad_id in enumerate(ids, start = 1): # call download routine for every id
LOG.info("Downloading %d/%d ads...", idx, len(ids))
exists = await ad_extractor.navigate_to_ad_page(ad_id)
if exists:
resolved = _download_selection.resolve_download_ad_activity(ad_id, published_ads_by_id)
if not resolved.owned:
# Foreign ad - expected for numeric IDs (can download any public ad)
LOG.warning("Ad id %d is not in your published profile ads. Saving downloaded ad as inactive.", ad_id)
await ad_extractor.download_ad(ad_id, active = resolved.active)
LOG.info("Downloaded ad with id %d", ad_id)
else:
LOG.error("The page with the id %d does not exist!", ad_id)
#############################
# main entry point
#############################

1
src/kleinanzeigen_bot/ad_loading.py
View File

@@ -37,7 +37,6 @@ from .utils.i18n import pluralize
from .utils.misc import ensure
LOG:Final[_loggers.Logger] = _loggers.get_logger(__name__)
LOG.setLevel(_loggers.INFO)
# --------------------------------------------------------------------------- #

136
src/kleinanzeigen_bot/delete_flow.py Normal file
View File

@@ -0,0 +1,136 @@
# SPDX-FileCopyrightText: © Sebastian Thomschke and contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
# SPDX-ArtifactOfProjectHomePage: https://github.com/Second-Hand-Friends/kleinanzeigen-bot/
"""Ad deletion browser workflow."""
from gettext import gettext as _
from typing import Any, Final, Literal
from . import ad_state as _ad_state
from . import published_ads
from .model.ad_model import Ad
from .published_ads import PublishedAd
from .utils import dicts as _dicts
from .utils import loggers as _loggers
from .utils.i18n import pluralize
from .utils.misc import ensure
from .utils.web_scraping_mixin import By, WebScrapingMixin
LOG:_loggers.Logger = _loggers.get_logger(__name__)
async def delete_ads(
web:WebScrapingMixin,
root_url:str,
after_delete:Literal["NONE", "RESET", "DISABLE"],
*,
delete_old_ads_by_title:bool,
ad_cfgs:list[tuple[str, Ad, dict[str, Any]]],
) -> None:
count = 0
deleted_count = 0
published_ads_list = await published_ads.fetch_published_ads(web, root_url)
for ad_file, ad_cfg, ad_cfg_orig in ad_cfgs:
count += 1
LOG.info("Processing %s/%s: '%s' from [%s]...", count, len(ad_cfgs), ad_cfg.title, ad_file)
# Record pre-delete id to detect whether delete_ad attempted a deletion.
# delete_ad clears ad_cfg.id when targets were found (Phase B ran),
# and preserves it on no-match early return.
id_before = ad_cfg.id
deleted = await delete_ad(web, root_url, ad_cfg, published_ads_list, delete_old_ads_by_title = delete_old_ads_by_title)
if deleted:
deleted_count += 1
# Apply after_delete policy only when a delete was actually attempted.
# Detection: True return (some 200), or id changed from non-None to None (all 404).
# When id was already None before the call, only True return is reliable;
# a False return could be no-match or title-match all-404, both treated as no cleanup.
delete_attempted = deleted or (id_before is not None and ad_cfg.id is None)
if delete_attempted and after_delete != "NONE":
if _ad_state.apply_after_delete_policy(ad_cfg, ad_cfg_orig, mode = after_delete):
_dicts.save_dict(ad_file, ad_cfg_orig)
await web.web_sleep()
LOG.info("############################################")
LOG.info("DONE: Deleted %s of %s", deleted_count, pluralize("ad", count))
LOG.info("############################################")
async def delete_ad(
web:WebScrapingMixin,
root_url:str,
ad_cfg:Ad,
published_ads_list:list[PublishedAd],
*,
delete_old_ads_by_title:bool,
) -> bool:
"""Delete an ad from the server.
Returns:
True if at least one delete request returned 200 (confirmed deleted).
False if no matching ads were found or all returned 404 (already gone).
Side effects:
Clears ``ad_cfg.id`` whenever a delete was attempted, regardless of
the server response (200 or 404). The old ID is stale in both cases.
Preserves ``ad_cfg.id`` when no deletion was attempted (early return).
"""
LOG.info("Deleting ad '%s' if already present...", ad_cfg.title)
# Phase A: Build set of IDs to delete, unified across both modes
ids_to_delete:set[int] = set()
if delete_old_ads_by_title:
for published_ad in published_ads_list:
raw_id = published_ad.get("id")
if raw_id is None:
LOG.debug("Skipping published ad with missing id: %r", published_ad.get("title"))
continue
try:
published_ad_id = int(raw_id)
except (ValueError, TypeError):
LOG.debug("Skipping published ad with invalid id: %r", raw_id)
continue
published_ad_title = published_ad.get("title", "")
if ad_cfg.id == published_ad_id or ad_cfg.title == published_ad_title:
LOG.debug(" -> matched ad %s '%s' for deletion", published_ad_id, published_ad_title)
ids_to_delete.add(published_ad_id)
elif ad_cfg.id is not None:
ids_to_delete.add(ad_cfg.id)
# Early return if nothing to delete — skip page open, CSRF fetch, and sleep
if not ids_to_delete:
LOG.info(" -> SKIPPED: no published ad matched '%s' for deletion", ad_cfg.title)
return False
# Phase B: Open manage-ads page, fetch CSRF token, execute deletions
await web.web_open(f"{root_url}/m-meine-anzeigen.html")
csrf_token_elem = await web.web_find(By.CSS_SELECTOR, "meta[name=_csrf]")
csrf_token = csrf_token_elem.attrs.get("content")
ensure(csrf_token is not None and isinstance(csrf_token, str) and csrf_token.strip(), _("Expected CSRF Token not found in HTML content!"))
HTTP_OK:Final = 200
deleted = False
for target_id in ids_to_delete:
LOG.debug(" -> deleting ad %s...", target_id)
response = await web.web_request(
url = f"{root_url}/m-anzeigen-loeschen.json?ids={target_id}",
method = "POST",
headers = {"x-csrf-token": str(csrf_token)},
valid_response_codes = [200, 404],
)
if response["statusCode"] == HTTP_OK:
deleted = True
LOG.info(" -> SUCCESS: deleted ad '%s' (ID: %s)", ad_cfg.title, target_id)
else:
LOG.warning(" -> ad %s not found (status %s), may have been removed already", target_id, response["statusCode"])
await web.web_sleep()
# Clear ad_cfg.id whenever a delete was attempted — the old ID is stale
# regardless of whether the server returned 200 (deleted) or 404 (already gone).
ad_cfg.id = None
return deleted

207
src/kleinanzeigen_bot/download_flow.py Normal file
View File

@@ -0,0 +1,207 @@
# SPDX-FileCopyrightText: © Sebastian Thomschke and contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
# SPDX-ArtifactOfProjectHomePage: https://github.com/Second-Hand-Friends/kleinanzeigen-bot/
"""Ad download browser workflow."""
from pathlib import Path
from typing import Any, Protocol
from . import download_selection as _download_selection
from . import extract, published_ads
from .model.ad_model import Ad
from .model.config_model import DEFAULT_DOWNLOAD_DIR, Config
from .published_ads import PublishedAd
from .utils import loggers as _loggers
from .utils import xdg_paths as _xdg_paths
from .utils.files import abspath
from .utils.i18n import pluralize
from .utils.web_scraping_mixin import WebScrapingMixin
class LoadAdsFunc(Protocol):
"""Protocol for callable that loads ads, matching ad_loading.load_ads signature."""
def __call__(self, *, ignore_inactive:bool = True, exclude_ads_with_id:bool = True) -> list[tuple[str, Ad, dict[str, Any]]]:
raise NotImplementedError
LOG:_loggers.Logger = _loggers.get_logger(__name__)
def resolve_download_dir(
config:Config,
config_file_path:str,
workspace:_xdg_paths.Workspace,
) -> Path:
"""Resolve the download directory from config and workspace.
Returns workspace.download_dir when config.download.dir is the literal
default; otherwise resolves the configured path (relative to config file
or absolute).
"""
trimmed_dir = config.download.dir.strip()
if trimmed_dir == DEFAULT_DOWNLOAD_DIR:
return workspace.download_dir
return Path(abspath(trimmed_dir, relative_to = str(Path(config_file_path).parent))).resolve()
async def _download_ad_with_resolved_state(
ad_extractor:extract.AdExtractor,
ad_id:int,
published_ads_by_id:dict[int, PublishedAd],
) -> None:
"""Download an ad with proper active state resolution and logging.
Resolves the ad's activity state from the published profile, logs appropriately
based on the resolution result, and initiates the download with the resolved state.
This function centralizes the resolution + logging + download logic used by
the "all" and "new" selectors.
Args:
ad_extractor: The AdExtractor instance to use for downloading.
ad_id: The ad ID to download.
published_ads_by_id: Dict mapping ad IDs to published ad data from API.
Note:
The numeric selector does NOT use this helper because it has different
warning message semantics (foreign ads are expected, not anomalies).
"""
resolved = _download_selection.resolve_download_ad_activity(ad_id, published_ads_by_id)
if not resolved.owned:
# Ad not in user's published profile - unexpected for "all"/"new" selectors
# since these only list the user's own ads from the overview page
LOG.warning("Ad %d found in overview but not in published profile. Saving as inactive.", ad_id)
elif not resolved.active:
# Ad is in published profile but not in active state (paused, inactive, etc.)
published_ad = published_ads_by_id.get(ad_id, {})
LOG.debug("Ad %d has state '%s'. Saving as inactive.", ad_id, published_ad.get("state", "unknown"))
await ad_extractor.download_ad(ad_id, active = resolved.active)
async def download_ads(
web:WebScrapingMixin,
config:Config,
config_file_path:str,
workspace:_xdg_paths.Workspace,
ads_selector:str,
*,
load_ads_func:LoadAdsFunc,
root_url:str,
) -> None:
"""
Determines which download mode was chosen with the arguments, and calls the specified download routine.
This downloads either all, only unsaved(new), or specific ads given by ID.
"""
# Normalize comma-separated keyword selectors; set deduplication collapses "new,new" → {"new"}
selector_tokens = {s.strip() for s in ads_selector.split(",")}
if "all" in selector_tokens:
effective_selector = "all"
elif len(selector_tokens) == 1:
effective_selector = next(iter(selector_tokens)) # e.g. "new,new" → "new"
else:
effective_selector = ads_selector # numeric IDs: "123,456" — unchanged
# Fetch published ads once from manage-ads JSON to avoid repetitive API calls during extraction
# Build lookup dict inline and pass directly to extractor (no cache abstraction needed)
LOG.info("Fetching ad metadata (status, expiry dates)...")
published_ads_list = await published_ads.fetch_published_ads(web, root_url, strict = _download_selection.is_numeric_ids_selector(effective_selector))
published_ads_by_id:dict[int, PublishedAd] = {}
for published_ad in published_ads_list:
try:
ad_id = published_ad.get("id")
if ad_id is not None:
published_ads_by_id[int(ad_id)] = published_ad
except (ValueError, TypeError):
LOG.warning("Skipping ad with non-numeric id: %s", published_ad.get("id"))
LOG.info("Loaded metadata for %s published ads.", len(published_ads_by_id))
download_dir = resolve_download_dir(config, config_file_path, workspace)
_xdg_paths.ensure_directory(download_dir, "downloaded ads directory")
LOG.info("Ads download directory: %s", download_dir)
ad_extractor = extract.AdExtractor(web.browser, config, download_dir, published_ads_by_id = published_ads_by_id)
if effective_selector in {"all", "new"}: # explore ads overview for these two modes
LOG.info("Scanning ad overview for navigation URLs...")
own_ad_urls = await ad_extractor.extract_own_ads_urls()
LOG.info("Found %s.", pluralize("ad URL", len(own_ad_urls)))
if effective_selector == "all": # download all of your ads
LOG.info("Starting download of all ads...")
valid_ad_refs:list[tuple[str, int]] = []
for ad_url in own_ad_urls:
ad_id = ad_extractor.extract_ad_id_from_ad_url(ad_url)
if ad_id == -1:
# Skip ads with invalid URLs (warning already logged by extract_ad_id_from_ad_url)
continue
valid_ad_refs.append((ad_url, ad_id))
success_count = 0
# call download function for each ad page
for idx, (ad_url, ad_id) in enumerate(valid_ad_refs, start = 1):
LOG.info("Downloading %d/%d ads...", idx, len(valid_ad_refs))
if await ad_extractor.navigate_to_ad_page(ad_url):
await _download_ad_with_resolved_state(ad_extractor, ad_id, published_ads_by_id)
success_count += 1
LOG.info("%d of %d ads were downloaded from your profile.", success_count, len(valid_ad_refs))
elif effective_selector == "new": # download only unsaved ads
# check which ads already saved
saved_ad_ids:set[int] = set()
ads = load_ads_func(ignore_inactive = False, exclude_ads_with_id = False) # do not skip because of existing IDs
for ad in ads:
saved_ad_id = ad[1].id
if saved_ad_id is None:
LOG.debug("Skipping saved ad without id (likely unpublished or manually created): %s", ad[0])
continue
saved_ad_ids.add(int(saved_ad_id))
# determine ad IDs from links
ad_id_by_url = {url: ad_extractor.extract_ad_id_from_ad_url(url) for url in own_ad_urls}
LOG.info("Starting download of not yet downloaded ads...")
ads_to_download:list[tuple[str, int]] = []
for ad_url, ad_id in ad_id_by_url.items():
# Skip ads with invalid URLs (warning already logged by extract_ad_id_from_ad_url)
if ad_id == -1:
continue
# check if ad with ID already saved
if ad_id in saved_ad_ids:
LOG.info("The ad with id %d has already been saved.", ad_id)
continue
ads_to_download.append((ad_url, ad_id))
new_count = 0
for idx, (ad_url, ad_id) in enumerate(ads_to_download, start = 1):
LOG.info("Downloading %d/%d ads...", idx, len(ads_to_download))
if await ad_extractor.navigate_to_ad_page(ad_url):
await _download_ad_with_resolved_state(ad_extractor, ad_id, published_ads_by_id)
new_count += 1
LOG.info("%s were downloaded from your profile.", pluralize("new ad", new_count))
elif _download_selection.is_numeric_ids_selector(effective_selector): # download ad(s) with specific id(s)
ids = [int(n) for n in effective_selector.split(",")]
LOG.info("Starting download of ad(s) with the id(s):")
LOG.info(" | ".join([str(ad_id) for ad_id in ids]))
for idx, ad_id in enumerate(ids, start = 1): # call download routine for every id
LOG.info("Downloading %d/%d ads...", idx, len(ids))
exists = await ad_extractor.navigate_to_ad_page(ad_id)
if exists:
resolved = _download_selection.resolve_download_ad_activity(ad_id, published_ads_by_id)
if not resolved.owned:
# Foreign ad - expected for numeric IDs (can download any public ad)
LOG.warning("Ad id %d is not in your published profile ads. Saving downloaded ad as inactive.", ad_id)
await ad_extractor.download_ad(ad_id, active = resolved.active)
LOG.info("Downloaded ad with id %d", ad_id)
else:
LOG.error("The page with the id %d does not exist!", ad_id)
else:
LOG.error("Invalid ads selector: %s. Use 'all', 'new', or comma-separated numeric IDs.", effective_selector)

153
src/kleinanzeigen_bot/extend_flow.py Normal file
View File

@@ -0,0 +1,153 @@
# SPDX-FileCopyrightText: © Sebastian Thomschke and contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
# SPDX-ArtifactOfProjectHomePage: https://github.com/Second-Hand-Friends/kleinanzeigen-bot/
"""Ad extension browser workflow."""
from __future__ import annotations
from datetime import datetime
from typing import TYPE_CHECKING, Any
from . import published_ads
from .published_ads import ad_matches_id
from .utils import dicts as _dicts
if TYPE_CHECKING:
from .model.ad_model import Ad
from .published_ads import PublishedAd
from .utils import loggers as _loggers
from .utils import misc as _misc
from .utils.i18n import pluralize
from .utils.web_scraping_mixin import By, WebScrapingMixin
LOG:_loggers.Logger = _loggers.get_logger(__name__)
async def extend_ads(
web:WebScrapingMixin,
root_url:str,
ad_cfgs:list[tuple[str, Ad, dict[str, Any]]],
) -> None:
"""Extends ads that are close to expiry."""
# Fetch currently published ads from API
published_ads_list = await published_ads.fetch_published_ads(web, root_url)
# Filter ads that need extension
ads_to_extend:list[tuple[str, Ad, dict[str, Any]]] = []
for ad_file, ad_cfg, ad_cfg_orig in ad_cfgs:
# Skip unpublished ads (no ID)
if ad_cfg.id is None:
LOG.info(" -> SKIPPED: ad '%s' is not published yet", ad_cfg.title)
continue
published_ad:PublishedAd | None = next(
(ad for ad in published_ads_list if ad_matches_id(ad, ad_cfg.id)), None
)
if not published_ad:
LOG.warning(" -> SKIPPED: ad '%s' (ID: %s) not found in published ads", ad_cfg.title, ad_cfg.id)
continue
# Skip non-active ads
if published_ad.get("state") != "active":
LOG.info(" -> SKIPPED: ad '%s' is not active (state: %s)", ad_cfg.title, published_ad.get("state"))
continue
# Check if ad is within 8-day extension window using API's endDate
end_date_str = published_ad.get("endDate")
if not end_date_str:
LOG.warning(" -> SKIPPED: ad '%s' has no endDate in API response", ad_cfg.title)
continue
# Intentionally parsing naive datetime from kleinanzeigen API's German date format, timezone not relevant for date-only comparison
try:
end_date = datetime.strptime(end_date_str, "%d.%m.%Y") # noqa: DTZ007
except (ValueError, TypeError):
LOG.warning(" -> SKIPPED: ad '%s' has invalid endDate format: %s", ad_cfg.title, end_date_str)
continue
days_until_expiry = (end_date.date() - _misc.now().date()).days
# Magic value 8 is kleinanzeigen.de's platform policy: extensions only possible within 8 days of expiry
if days_until_expiry <= 8: # noqa: PLR2004
LOG.info(" -> ad '%s' expires in %d days, will extend", ad_cfg.title, days_until_expiry)
ads_to_extend.append((ad_file, ad_cfg, ad_cfg_orig))
else:
LOG.info(" -> SKIPPED: ad '%s' expires in %d days (can only extend within 8 days)", ad_cfg.title, days_until_expiry)
if not ads_to_extend:
LOG.info("No ads need extension at this time.")
LOG.info("############################################")
LOG.info("DONE: No ads extended.")
LOG.info("############################################")
return
# Process extensions
success_count = 0
for idx, (ad_file, ad_cfg, ad_cfg_orig) in enumerate(ads_to_extend, start = 1):
LOG.info("Processing %s/%s: '%s' from [%s]...", idx, len(ads_to_extend), ad_cfg.title, ad_file)
if await _extend_ad(web, root_url, ad_file, ad_cfg, ad_cfg_orig):
success_count += 1
await web.web_sleep()
LOG.info("############################################")
LOG.info("DONE: Extended %s", pluralize("ad", success_count))
LOG.info("############################################")
async def _extend_ad(
web:WebScrapingMixin,
root_url:str,
ad_file:str,
ad_cfg:Ad,
ad_cfg_orig:dict[str, Any],
) -> bool:
"""Extends a single ad listing."""
LOG.info("Extending ad '%s' (ID: %s)...", ad_cfg.title, ad_cfg.id)
try:
# Navigate to ad management page and find extend button across all pages
extend_button_xpath = f'//li[@data-adid="{ad_cfg.id}"]//button[contains(., "Verlängern")]'
async def find_and_click_extend_button(page_num:int) -> bool:
"""Try to find and click extend button on current page."""
try:
extend_button = await web.web_find(By.XPATH, extend_button_xpath, timeout = web.timeout("quick_dom"))
LOG.info("Found extend button on page %s", page_num)
await extend_button.click()
return True # Success - stop pagination
except TimeoutError:
LOG.debug("Extend button not found on page %s", page_num)
return False # Continue to next page
success = await web.navigate_paginated_ad_overview(find_and_click_extend_button, page_url = f"{root_url}/m-meine-anzeigen.html")
if not success:
LOG.error(" -> FAILED: Could not find extend button for ad ID %s", ad_cfg.id)
return False
# Handle confirmation dialog
# After clicking "Verlängern", a dialog appears with:
# - Title: "Vielen Dank!"
# - Message: "Deine Anzeige ... wurde erfolgreich verlängert."
# - Paid bump-up option (skipped by closing dialog)
# Simply close the dialog with the X button (aria-label="Schließen")
try:
dialog_close_timeout = web.timeout("quick_dom")
await web.web_click(By.CSS_SELECTOR, 'button[aria-label="Schließen"]', timeout = dialog_close_timeout)
LOG.debug(" -> Closed confirmation dialog")
except TimeoutError:
LOG.warning(" -> No confirmation dialog found, extension may have completed directly")
# Update metadata in YAML file
# Update updated_on to track when ad was extended
ad_cfg_orig["updated_on"] = _misc.now().isoformat(timespec = "seconds")
_dicts.save_dict(ad_file, ad_cfg_orig)
LOG.info(" -> SUCCESS: ad extended with ID %s", ad_cfg.id)
return True
except TimeoutError as ex:
LOG.error(" -> FAILED: Timeout while extending ad '%s': %s", ad_cfg.title, ex)
return False
except OSError as ex:
LOG.error(" -> FAILED: Could not persist extension for ad '%s': %s", ad_cfg.title, ex)
return False

2
src/kleinanzeigen_bot/extract.py
View File

@@ -531,7 +531,7 @@ class AdExtractor(WebScrapingMixin):
LOG.exception("Error extracting refs on page %s: %s", page_num, e)
return False # Continue to next page
await self._navigate_paginated_ad_overview(extract_page_refs)
await self.navigate_paginated_ad_overview(extract_page_refs)
if not refs:
LOG.warning("No ad URLs were extracted.")

183
src/kleinanzeigen_bot/published_ads.py Normal file
View File

@@ -0,0 +1,183 @@
# SPDX-FileCopyrightText: © Sebastian Thomschke and contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
# SPDX-ArtifactOfProjectHomePage: https://github.com/Second-Hand-Friends/kleinanzeigen-bot/
"""Published ads fetching with API pagination."""
import json
from gettext import gettext as _
from typing import Any, Final, TypeAlias
from .utils import loggers as _loggers
from .utils import misc as _misc
from .utils.exceptions import KleinanzeigenBotError
from .utils.web_scraping_mixin import WebScrapingMixin
PublishedAd:TypeAlias = dict[str, Any]
"""A raw published ad entry from the Kleinanzeigen manage-ads JSON API."""
def ad_matches_id(ad:PublishedAd, target_id:int | None) -> bool:
"""Check if a published ad matches the given target ID.
Normalizes API IDs (which may be ``str`` or ``int``) for safe comparison.
Returns ``False`` when ``target_id`` is ``None`` or the ad's ``id`` key
is missing, unparseable, or of an unexpected type.
"""
if target_id is None:
return False
raw_id = ad.get("id")
if raw_id is None:
return False
try:
return int(raw_id) == target_id
except (TypeError, ValueError):
return False
LOG:Final = _loggers.get_logger(__name__)
class PublishedAdsFetchIncompleteError(KleinanzeigenBotError):
"""Raised when published ads cannot be fetched completely for ownership-critical operations."""
async def fetch_published_ads(
web:WebScrapingMixin,
root_url:str,
*,
strict:bool = False,
) -> list[PublishedAd]:
"""Fetch all published ads, handling API pagination.
Args:
web: A WebScrapingMixin instance for making web requests.
root_url: The base URL of the Kleinanzeigen site.
strict: If True, raise PublishedAdsFetchIncompleteError when pagination data is incomplete.
Returns:
List of all published ads across all pages.
"""
ads:list[PublishedAd] = []
page = 1
MAX_PAGE_LIMIT:Final[int] = 100
SNIPPET_LIMIT:Final[int] = 500
def _handle_incomplete_fetch(template:str, *args:Any, cause:Exception | None = None) -> None:
if strict:
raise PublishedAdsFetchIncompleteError(_(template) % args) from cause
while True:
# Safety check: don't paginate beyond reasonable limit
if page > MAX_PAGE_LIMIT:
LOG.warning("Stopping pagination after %s pages to avoid infinite loop", MAX_PAGE_LIMIT)
_handle_incomplete_fetch("Stopping pagination after %s pages to avoid infinite loop", MAX_PAGE_LIMIT)
break
try:
response = await web.web_request(f"{root_url}/m-meine-anzeigen-verwalten.json?sort=DEFAULT&pageNum={page}")
except TimeoutError as ex:
LOG.warning("Pagination request failed on page %s: %s", page, ex)
_handle_incomplete_fetch("Pagination request failed on page %s: %s", page, ex, cause = ex)
break
if not isinstance(response, dict):
LOG.warning("Unexpected pagination response type on page %s: %s", page, type(response).__name__)
_handle_incomplete_fetch("Unexpected pagination response type on page %s: %s", page, type(response).__name__)
break
content = response.get("content", "")
if isinstance(content, bytearray):
content = bytes(content)
if isinstance(content, bytes):
content = content.decode("utf-8", errors = "replace")
if not isinstance(content, str):
LOG.warning("Unexpected response content type on page %s: %s", page, type(content).__name__)
_handle_incomplete_fetch("Unexpected response content type on page %s: %s", page, type(content).__name__)
break
try:
json_data = json.loads(content)
except (json.JSONDecodeError, TypeError) as ex:
if not content:
LOG.warning("Empty JSON response content on page %s", page)
_handle_incomplete_fetch("Empty JSON response content on page %s", page, cause = ex)
break
snippet = content[:SNIPPET_LIMIT] + ("..." if len(content) > SNIPPET_LIMIT else "")
LOG.warning("Failed to parse JSON response on page %s: %s (content: %s)", page, ex, snippet)
_handle_incomplete_fetch("Failed to parse JSON response on page %s: %s (content: %s)", page, ex, snippet, cause = ex)
break
if not isinstance(json_data, dict):
snippet = content[:SNIPPET_LIMIT] + ("..." if len(content) > SNIPPET_LIMIT else "")
LOG.warning("Unexpected JSON payload on page %s (content: %s)", page, snippet)
_handle_incomplete_fetch("Unexpected JSON payload on page %s (content: %s)", page, snippet)
break
page_ads = json_data.get("ads", [])
if not isinstance(page_ads, list):
preview = str(page_ads)
if len(preview) > SNIPPET_LIMIT:
preview = preview[:SNIPPET_LIMIT] + "..."
LOG.warning("Unexpected 'ads' type on page %s: %s value: %s", page, type(page_ads).__name__, preview)
_handle_incomplete_fetch("Unexpected 'ads' type on page %s: %s value: %s", page, type(page_ads).__name__, preview)
break
filtered_page_ads:list[PublishedAd] = []
rejected_count = 0
rejected_preview:str | None = None
for entry in page_ads:
if isinstance(entry, dict) and "id" in entry and "state" in entry:
filtered_page_ads.append(entry)
continue
rejected_count += 1
if rejected_preview is None:
rejected_preview = repr(entry)
if rejected_count > 0:
preview = rejected_preview or "<none>"
if len(preview) > SNIPPET_LIMIT:
preview = preview[:SNIPPET_LIMIT] + "..."
LOG.warning("Filtered %s malformed ad entries on page %s (sample: %s)", rejected_count, page, preview)
_handle_incomplete_fetch("Filtered %s malformed ad entries on page %s (sample: %s)", rejected_count, page, preview)
ads.extend(filtered_page_ads)
paging = json_data.get("paging")
if not isinstance(paging, dict):
LOG.debug("No paging dict found on page %s, assuming single page", page)
_handle_incomplete_fetch("No paging dict found on page %s", page)
break
# Use only real API fields (confirmed from production data)
current_page_num = _misc.coerce_page_number(paging.get("pageNum"))
total_pages = _misc.coerce_page_number(paging.get("last"))
if current_page_num is None:
LOG.warning("Invalid 'pageNum' in paging info: %s, stopping pagination", paging.get("pageNum"))
_handle_incomplete_fetch("Invalid 'pageNum' in paging info: %s, stopping pagination", paging.get("pageNum"))
break
# Stop if reached last page (only when API provides 'last')
if total_pages is not None and current_page_num >= total_pages:
LOG.info("Reached last page %s of %s, stopping pagination", current_page_num, total_pages)
break
# Safety: stop if no ads returned
if len(page_ads) == 0:
LOG.info("No ads found on page %s, stopping pagination", page)
break
LOG.debug("Page %s: fetched %s ads (numFound=%s)", page, len(page_ads), paging.get("numFound"))
# Use API's next field for navigation (more robust than our counter)
next_page = _misc.coerce_page_number(paging.get("next"))
if next_page is None:
if total_pages is not None:
LOG.warning("Invalid 'next' page value in paging info: %s, stopping pagination", paging.get("next"))
_handle_incomplete_fetch("Invalid 'next' page value in paging info: %s, stopping pagination", paging.get("next"))
else:
LOG.debug("No 'next' in paging on page %s, assuming last page", page)
break
page = next_page
return ads

167
src/kleinanzeigen_bot/resources/translations.de.yaml
View File

@@ -58,21 +58,6 @@ kleinanzeigen_bot/__init__.py:
_workspace_or_raise:
"Workspace must be resolved before command execution": "Arbeitsbereich muss vor der Befehlsausführung aufgelöst werden"
_fetch_published_ads:
"Empty JSON response content on page %s": "Leerer JSON-Antwortinhalt auf Seite %s"
"Failed to parse JSON response on page %s: %s (content: %s)": "Fehler beim Parsen der JSON-Antwort auf Seite %s: %s (Inhalt: %s)"
"Stopping pagination after %s pages to avoid infinite loop": "Stoppe die Seitenaufschaltung nach %s Seiten, um eine Endlosschleife zu vermeiden"
"Pagination request failed on page %s: %s": "Seitenabfrage auf Seite %s fehlgeschlagen: %s"
"Unexpected pagination response type on page %s: %s": "Unerwarteter Typ der Paginierungsantwort auf Seite %s: %s"
"Unexpected response content type on page %s: %s": "Unerwarteter Antwortinhalt-Typ auf Seite %s: %s"
"Unexpected JSON payload on page %s (content: %s)": "Unerwartete JSON-Antwort auf Seite %s (Inhalt: %s)"
"Unexpected 'ads' type on page %s: %s value: %s": "Unerwarteter 'ads'-Typ auf Seite %s: %s Wert: %s"
"Filtered %s malformed ad entries on page %s (sample: %s)": "%s fehlerhafte Anzeigen-Einträge auf Seite %s gefiltert (Beispiel: %s)"
"Reached last page %s of %s, stopping pagination": "Letzte Seite %s von %s erreicht, beende Paginierung"
"No ads found on page %s, stopping pagination": "Keine Anzeigen auf Seite %s gefunden, beende Paginierung"
"Invalid 'next' page value in paging info: %s, stopping pagination": "Ungültiger 'next'-Seitenwert in Paginierungsinfo: %s, beende Paginierung"
"Invalid 'pageNum' in paging info: %s, stopping pagination": "Ungültiger 'pageNum'-Wert in Paginierungsinfo: %s, beende Paginierung"
check_and_wait_for_captcha:
"# Captcha present! Please solve the captcha.": "# Captcha vorhanden! Bitte lösen Sie das Captcha."
"Captcha recognized - auto-restart enabled, abort run...": "Captcha erkannt - Auto-Neustart aktiviert, Durchlauf wird beendet..."
@@ -125,42 +110,6 @@ kleinanzeigen_bot/__init__.py:
handle_after_login_logic:
"Handling GDPR disclaimer...": "Verarbeite DSGVO-Hinweis..."
delete_ads:
"Processing %s/%s: '%s' from [%s]...": "Verarbeite %s/%s: '%s' von [%s]..."
"DONE: Deleted %s of %s": "FERTIG: %s von %s gelöscht"
"ad": "Anzeige"
delete_ad:
"Deleting ad '%s' if already present...": "Lösche Anzeige '%s', falls bereits vorhanden..."
"Expected CSRF Token not found in HTML content!": "Erwartetes CSRF-Token wurde im HTML-Inhalt nicht gefunden!"
" -> SKIPPED: no published ad matched '%s' for deletion": " -> ÜBERSPRUNGEN: Keine veröffentlichte Anzeige '%s' zum Löschen gefunden"
" -> SUCCESS: deleted ad '%s' (ID: %s)": " -> ERFOLG: Anzeige '%s' (ID: %s) gelöscht"
" -> ad %s not found (status %s), may have been removed already": " -> Anzeige %s nicht gefunden (Status %s), möglicherweise bereits entfernt"
extend_ads:
"No ads need extension at this time.": "Keine Anzeigen müssen derzeit verlängert werden."
"DONE: No ads extended.": "FERTIG: Keine Anzeigen verlängert."
"DONE: Extended %s": "FERTIG: %s verlängert"
"ad": "Anzeige"
" -> SKIPPED: ad '%s' is not published yet": " -> ÜBERSPRUNGEN: Anzeige '%s' ist noch nicht veröffentlicht"
" -> SKIPPED: ad '%s' (ID: %s) not found in published ads": " -> ÜBERSPRUNGEN: Anzeige '%s' (ID: %s) nicht gefunden"
" -> SKIPPED: ad '%s' is not active (state: %s)": " -> ÜBERSPRUNGEN: Anzeige '%s' ist nicht aktiv (Status: %s)"
" -> SKIPPED: ad '%s' has no endDate in API response": " -> ÜBERSPRUNGEN: Anzeige '%s' hat kein Ablaufdatum in API-Antwort"
" -> ad '%s' expires in %d days, will extend": " -> Anzeige '%s' läuft in %d Tagen ab, wird verlängert"
" -> SKIPPED: ad '%s' expires in %d days (can only extend within 8 days)": " -> ÜBERSPRUNGEN: Anzeige '%s' läuft in %d Tagen ab (Verlängern nur innerhalb von 8 Tagen möglich)"
"Processing %s/%s: '%s' from [%s]...": "Verarbeite %s/%s: '%s' aus [%s]..."
extend_ad:
"Extending ad '%s' (ID: %s)...": "Verlängere Anzeige '%s' (ID: %s)..."
" -> FAILED: Could not find extend button for ad ID %s": " -> FEHLER: 'Verlängern'-Button für Anzeigen-ID %s nicht gefunden"
" -> No confirmation dialog found, extension may have completed directly": " -> Kein Bestätigungsdialog gefunden"
" -> SUCCESS: ad extended with ID %s": " -> ERFOLG: Anzeige mit ID %s verlängert"
" -> FAILED: Timeout while extending ad '%s': %s": " -> FEHLER: Zeitüberschreitung beim Verlängern der Anzeige '%s': %s"
" -> FAILED: Could not persist extension for ad '%s': %s": " -> FEHLER: Verlängerung der Anzeige '%s' konnte nicht gespeichert werden: %s"
find_and_click_extend_button:
"Found extend button on page %s": "'Verlängern'-Button auf Seite %s gefunden"
publish_ads:
"Processing %s/%s: '%s' from [%s]...": "Verarbeite %s/%s: '%s' von [%s]..."
"Skipping because ad is reserved": "Überspringen, da Anzeige reserviert ist"
@@ -279,29 +228,6 @@ kleinanzeigen_bot/__init__.py:
"Attribute field '%s' seems to be a Combobox (i.e. text input with filtering dropdown)...": "Attributfeld '%s' scheint eine Combobox zu sein (d.h. Texteingabefeld mit Dropdown-Filter)..."
"Condition dialog not available, falling back to generic attribute handler for [%s]...": "Zustandsdialog nicht verfügbar, falle auf generischen Attribut-Handler für [%s] zurück..."
download_ads:
"Ads download directory: %s": "Anzeigen-Download-Verzeichnis: %s"
"Fetching ad metadata (status, expiry dates)...": "Lade Anzeigen-Metadaten (Status, Ablaufdaten)..."
"Loaded metadata for %s published ads.": "Metadaten für %s veröffentlichte Anzeigen geladen."
"Scanning ad overview for navigation URLs...": "Suche Navigations-URLs in Anzeigenübersicht..."
"Found %s.": "%s gefunden."
"ad URL": "Anzeigen-URL"
"Starting download of all ads...": "Starte den Download aller Anzeigen..."
"Downloading %d/%d ads...": "Lade Anzeige %d/%d herunter..."
"%d of %d ads were downloaded from your profile.": "%d von %d Anzeigen wurden aus Ihrem Profil heruntergeladen."
"Starting download of not yet downloaded ads...": "Starte den Download noch nicht heruntergeladener Anzeigen..."
"Skipping ad with non-numeric id: %s": "Überspringe Anzeige mit nicht-numerischer ID: %s"
"The ad with id %d has already been saved.": "Die Anzeige mit der ID %d wurde bereits gespeichert."
"%s were downloaded from your profile.": "%s wurden aus Ihrem Profil heruntergeladen."
"new ad": "neue Anzeige"
"Starting download of ad(s) with the id(s):": "Starte Download der Anzeige(n) mit den ID(s):"
"Ad id %d is not in your published profile ads. Saving downloaded ad as inactive.": "Anzeigen-ID %d ist nicht in Ihren veröffentlichten Profilanzeigen. Speichere die heruntergeladene Anzeige als inaktiv."
"Downloaded ad with id %d": "Anzeige mit der ID %d heruntergeladen"
"The page with the id %d does not exist!": "Die Seite mit der ID %d existiert nicht!"
_download_ad_with_resolved_state:
"Ad %d found in overview but not in published profile. Saving as inactive.": "Anzeige %d wurde in der Übersicht gefunden, aber nicht im veröffentlichten Profil. Speichere als inaktiv."
run:
"DONE: No configuration errors found.": "FERTIG: Keine Konfigurationsfehler gefunden."
"DONE: No active ads found.": "FERTIG: Keine aktiven Anzeigen gefunden."
@@ -334,6 +260,97 @@ kleinanzeigen_bot/__init__.py:
"Unknown shipping option(s), please refer to the documentation/README: %s": "Unbekannte Versandoption(en), bitte lies die Dokumentation/das README: %s"
"You can only specify shipping options for one package size!": "Du kannst nur Versandoptionen für eine Paketgröße angeben!"
#################################################
kleinanzeigen_bot/published_ads.py:
#################################################
fetch_published_ads:
"Empty JSON response content on page %s": "Leerer JSON-Antwortinhalt auf Seite %s"
"Failed to parse JSON response on page %s: %s (content: %s)": "Fehler beim Parsen der JSON-Antwort auf Seite %s: %s (Inhalt: %s)"
"Stopping pagination after %s pages to avoid infinite loop": "Stoppe die Seitenaufschaltung nach %s Seiten, um eine Endlosschleife zu vermeiden"
"Pagination request failed on page %s: %s": "Seitenabfrage auf Seite %s fehlgeschlagen: %s"
"Unexpected pagination response type on page %s: %s": "Unerwarteter Typ der Paginierungsantwort auf Seite %s: %s"
"Unexpected response content type on page %s: %s": "Unerwarteter Antwortinhalt-Typ auf Seite %s: %s"
"Unexpected JSON payload on page %s (content: %s)": "Unerwartete JSON-Antwort auf Seite %s (Inhalt: %s)"
"Unexpected 'ads' type on page %s: %s value: %s": "Unerwarteter 'ads'-Typ auf Seite %s: %s Wert: %s"
"Filtered %s malformed ad entries on page %s (sample: %s)": "%s fehlerhafte Anzeigen-Einträge auf Seite %s gefiltert (Beispiel: %s)"
"Reached last page %s of %s, stopping pagination": "Letzte Seite %s von %s erreicht, beende Paginierung"
"No ads found on page %s, stopping pagination": "Keine Anzeigen auf Seite %s gefunden, beende Paginierung"
"Invalid 'next' page value in paging info: %s, stopping pagination": "Ungültiger 'next'-Seitenwert in Paginierungsinfo: %s, beende Paginierung"
"Invalid 'pageNum' in paging info: %s, stopping pagination": "Ungültiger 'pageNum'-Wert in Paginierungsinfo: %s, beende Paginierung"
#################################################
kleinanzeigen_bot/delete_flow.py:
#################################################
delete_ads:
"############################################": "############################################"
"Processing %s/%s: '%s' from [%s]...": "Verarbeite %s/%s: '%s' von [%s]..."
"DONE: Deleted %s of %s": "FERTIG: %s von %s gelöscht"
"ad": "Anzeige"
delete_ad:
"Deleting ad '%s' if already present...": "Lösche Anzeige '%s', falls bereits vorhanden..."
"Expected CSRF Token not found in HTML content!": "Erwartetes CSRF-Token wurde im HTML-Inhalt nicht gefunden!"
" -> SKIPPED: no published ad matched '%s' for deletion": " -> ÜBERSPRUNGEN: Keine veröffentlichte Anzeige '%s' zum Löschen gefunden"
" -> SUCCESS: deleted ad '%s' (ID: %s)": " -> ERFOLG: Anzeige '%s' (ID: %s) gelöscht"
" -> ad %s not found (status %s), may have been removed already": " -> Anzeige %s nicht gefunden (Status %s), möglicherweise bereits entfernt"
#################################################
kleinanzeigen_bot/extend_flow.py:
#################################################
extend_ads:
"############################################": "############################################"
"No ads need extension at this time.": "Keine Anzeigen müssen derzeit verlängert werden."
"DONE: No ads extended.": "FERTIG: Keine Anzeigen verlängert."
"DONE: Extended %s": "FERTIG: %s verlängert"
"ad": "Anzeige"
" -> SKIPPED: ad '%s' is not published yet": " -> ÜBERSPRUNGEN: Anzeige '%s' ist noch nicht veröffentlicht"
" -> SKIPPED: ad '%s' (ID: %s) not found in published ads": " -> ÜBERSPRUNGEN: Anzeige '%s' (ID: %s) nicht gefunden"
" -> SKIPPED: ad '%s' is not active (state: %s)": " -> ÜBERSPRUNGEN: Anzeige '%s' ist nicht aktiv (Status: %s)"
" -> SKIPPED: ad '%s' has no endDate in API response": " -> ÜBERSPRUNGEN: Anzeige '%s' hat kein Ablaufdatum in API-Antwort"
" -> SKIPPED: ad '%s' has invalid endDate format: %s": " -> ÜBERSPRUNGEN: Anzeige '%s' hat ungültiges Ablaufdatum-Format: %s"
" -> ad '%s' expires in %d days, will extend": " -> Anzeige '%s' läuft in %d Tagen ab, wird verlängert"
" -> SKIPPED: ad '%s' expires in %d days (can only extend within 8 days)": " -> ÜBERSPRUNGEN: Anzeige '%s' läuft in %d Tagen ab (Verlängern nur innerhalb von 8 Tagen möglich)"
"Processing %s/%s: '%s' from [%s]...": "Verarbeite %s/%s: '%s' aus [%s]..."
_extend_ad:
"Extending ad '%s' (ID: %s)...": "Verlängere Anzeige '%s' (ID: %s)..."
" -> FAILED: Could not find extend button for ad ID %s": " -> FEHLER: 'Verlängern'-Button für Anzeigen-ID %s nicht gefunden"
" -> No confirmation dialog found, extension may have completed directly": " -> Kein Bestätigungsdialog gefunden"
" -> SUCCESS: ad extended with ID %s": " -> ERFOLG: Anzeige mit ID %s verlängert"
" -> FAILED: Timeout while extending ad '%s': %s": " -> FEHLER: Zeitüberschreitung beim Verlängern der Anzeige '%s': %s"
" -> FAILED: Could not persist extension for ad '%s': %s": " -> FEHLER: Verlängerung der Anzeige '%s' konnte nicht gespeichert werden: %s"
find_and_click_extend_button:
"Found extend button on page %s": "'Verlängern'-Button auf Seite %s gefunden"
#################################################
kleinanzeigen_bot/download_flow.py:
#################################################
download_ads:
"Ads download directory: %s": "Anzeigen-Download-Verzeichnis: %s"
"Fetching ad metadata (status, expiry dates)...": "Lade Anzeigen-Metadaten (Status, Ablaufdaten)..."
"Loaded metadata for %s published ads.": "Metadaten für %s veröffentlichte Anzeigen geladen."
"Scanning ad overview for navigation URLs...": "Suche Navigations-URLs in Anzeigenübersicht..."
"Found %s.": "%s gefunden."
"ad URL": "Anzeigen-URL"
"Starting download of all ads...": "Starte den Download aller Anzeigen..."
"Downloading %d/%d ads...": "Lade Anzeige %d/%d herunter..."
"%d of %d ads were downloaded from your profile.": "%d von %d Anzeigen wurden aus Ihrem Profil heruntergeladen."
"Starting download of not yet downloaded ads...": "Starte den Download noch nicht heruntergeladener Anzeigen..."
"Skipping ad with non-numeric id: %s": "Überspringe Anzeige mit nicht-numerischer ID: %s"
"The ad with id %d has already been saved.": "Die Anzeige mit der ID %d wurde bereits gespeichert."
"%s were downloaded from your profile.": "%s wurden aus Ihrem Profil heruntergeladen."
"new ad": "neue Anzeige"
"Starting download of ad(s) with the id(s):": "Starte Download der Anzeige(n) mit den ID(s):"
"Ad id %d is not in your published profile ads. Saving downloaded ad as inactive.": "Anzeigen-ID %d ist nicht in Ihren veröffentlichten Profilanzeigen. Speichere die heruntergeladene Anzeige als inaktiv."
"Downloaded ad with id %d": "Anzeige mit der ID %d heruntergeladen"
"Invalid ads selector: %s. Use 'all', 'new', or comma-separated numeric IDs.": "Ungültiger Anzeigen-Selektor: %s. Verwende 'all', 'new', oder kommagetrennte numerische IDs."
"The page with the id %d does not exist!": "Die Seite mit der ID %d existiert nicht!"
"Skipping saved ad without id (likely unpublished or manually created): %s": "Überspringe gespeicherte Anzeige ohne ID (vermutlich nicht veröffentlicht oder manuell erstellt): %s"
_download_ad_with_resolved_state:
"Ad %d found in overview but not in published profile. Saving as inactive.": "Anzeige %d wurde in der Übersicht gefunden, aber nicht im veröffentlichten Profil. Speichere als inaktiv."
#################################################
kleinanzeigen_bot/ad_loading.py:
#################################################
@@ -670,7 +687,7 @@ kleinanzeigen_bot/utils/web_scraping_mixin.py:
"Combobox fallback verified: '%s' confirmed.": "Combobox-Fallback bestätigt: '%s' verifiziert."
"No matching option found in listbox for combobox '%s'. Falling back to ArrowDown + Enter.": "Keine passende Option in der Listbox für Combobox '%s' gefunden. Fallback auf ArrowDown + Enter."
_navigate_paginated_ad_overview:
navigate_paginated_ad_overview:
"Failed to open ad overview page at %s: timeout": "Fehler beim Öffnen der Anzeigenübersichtsseite unter %s: Zeitüberschreitung"
"Scroll timeout on page %s (non-critical, continuing)": "Zeitüberschreitung beim Scrollen auf Seite %s (nicht kritisch, wird fortgesetzt)"
"Page action timed out on page %s": "Seitenaktion hat auf Seite %s eine Zeitüberschreitung erreicht"

4
src/kleinanzeigen_bot/utils/exceptions.py
View File

@@ -23,10 +23,6 @@ class PublishSubmissionUncertainError(KleinanzeigenBotError):
super().__init__(reason)
class PublishedAdsFetchIncompleteError(KleinanzeigenBotError):
"""Raised when published ads cannot be fetched completely for ownership-critical operations."""
class CategoryResolutionError(KleinanzeigenBotError):
"""Raised when the ad's configured category cannot be resolved by publish/update flows.

22
src/kleinanzeigen_bot/utils/web_scraping_mixin.py
View File

@@ -169,7 +169,7 @@ class WebScrapingMixin:
self._default_timeout_config = TimeoutConfig()
return self._default_timeout_config
def _timeout(self, key:str = "default", override:float | None = None) -> float:
def timeout(self, key:str = "default", override:float | None = None) -> float:
"""
Return the base timeout (seconds) for a given key without applying multipliers.
"""
@@ -225,7 +225,7 @@ class WebScrapingMixin:
Execute an async callable with retry/backoff handling for TimeoutError.
"""
attempts = self._timeout_attempts()
configured_timeout = self._timeout(key, override)
configured_timeout = self.timeout(key, override)
loop = asyncio.get_running_loop()
for attempt in range(attempts):
@@ -866,7 +866,7 @@ class WebScrapingMixin:
"""
loop = asyncio.get_running_loop()
start_at = loop.time()
base_timeout = timeout if timeout is not None else self._timeout()
base_timeout = timeout if timeout is not None else self.timeout()
effective_timeout = self._effective_timeout(override = base_timeout) if apply_multiplier else base_timeout
while True:
@@ -1080,7 +1080,7 @@ class WebScrapingMixin:
:param timeout: timeout in seconds (defaults to quick_dom when omitted)
:raises Exception: non-timeout browser/runtime exceptions are propagated
"""
probe_timeout = self._timeout("quick_dom", timeout)
probe_timeout = self.timeout("quick_dom", timeout)
try:
return await self._web_find_once(selector_type, selector_value, probe_timeout, parent = parent)
except TimeoutError:
@@ -1245,7 +1245,7 @@ class WebScrapingMixin:
)
await asyncio.sleep(duration / 1_000)
async def _navigate_paginated_ad_overview(
async def navigate_paginated_ad_overview(
self,
page_action:Callable[[int], Awaitable[bool]],
page_url:str = "https://www.kleinanzeigen.de/m-meine-anzeigen.html",
@@ -1274,7 +1274,7 @@ class WebScrapingMixin:
return True
return False
success = await self._navigate_paginated_ad_overview(find_ad_callback)
success = await self.navigate_paginated_ad_overview(find_ad_callback)
"""
try:
await self.web_open(page_url)
@@ -1298,7 +1298,7 @@ class WebScrapingMixin:
def is_enabled_next_button(button:Element) -> bool:
return button.attrs.get("disabled") is None and str(button.attrs.get("aria-disabled", "")).lower() != "true"
pagination_timeout = self._timeout("pagination_initial")
pagination_timeout = self.timeout("pagination_initial")
try:
next_buttons = await self.web_find_all(By.CSS_SELECTOR, next_page_selector, timeout = pagination_timeout)
enabled_next_buttons = [btn for btn in next_buttons if is_enabled_next_button(btn)]
@@ -1329,7 +1329,7 @@ class WebScrapingMixin:
if not multi_page:
break
follow_up_timeout = self._timeout("pagination_follow_up")
follow_up_timeout = self.timeout("pagination_follow_up")
try:
possible_next_buttons = await self.web_find_all(By.CSS_SELECTOR, next_page_selector, timeout = follow_up_timeout)
next_button_element = None
@@ -1464,7 +1464,7 @@ class WebScrapingMixin:
:raises TimeoutError: when the input or matching dropdown option cannot be located
"""
if timeout is None:
timeout = self._timeout("default")
timeout = self.timeout("default")
input_field = await self.web_find(selector_type, selector_value, timeout = timeout)
@@ -1490,7 +1490,7 @@ class WebScrapingMixin:
else:
LOG.debug("Combobox input field does not have 'aria-controls'. Trying listbox lookup by role.")
try:
dropdown_elem = await self.web_find(By.CSS_SELECTOR, '[role="listbox"]', timeout = self._timeout("quick_dom"))
dropdown_elem = await self.web_find(By.CSS_SELECTOR, '[role="listbox"]', timeout = self.timeout("quick_dom"))
except TimeoutError:
LOG.debug("No listbox found for combobox. Falling back to ArrowDown + Enter key confirmation.")
await self._dispatch_arrow_down_and_enter(input_field)
@@ -1591,7 +1591,7 @@ class WebScrapingMixin:
attribute handling — see GitHub issue #930.
"""
if timeout is None:
timeout = self._timeout("default")
timeout = self.timeout("default")
await self.web_click(By.ID, elem_id, timeout = timeout)
listbox_id = f"{elem_id}-menu"

407
tests/unit/test_delete_flow.py Normal file
View File

@@ -0,0 +1,407 @@
# SPDX-FileCopyrightText: © Jens Bergmann and contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
# SPDX-ArtifactOfProjectHomePage: https://github.com/Second-Hand-Friends/kleinanzeigen-bot/
"""Tests for ad deletion functionality."""
import copy
from pathlib import Path
from typing import Any
from unittest.mock import AsyncMock, patch
import pytest
from kleinanzeigen_bot import KleinanzeigenBot, delete_flow
from kleinanzeigen_bot.model.ad_model import Ad
@pytest.fixture
def base_ad_config() -> dict[str, Any]:
"""Provide a base ad configuration that can be used across tests."""
return {
"id": None,
"title": "Test Title",
"description": "Test Description",
"type": "OFFER",
"price_type": "FIXED",
"price": 100,
"shipping_type": "SHIPPING",
"shipping_options": [],
"category": "160",
"special_attributes": {},
"sell_directly": False,
"images": [],
"active": True,
"republication_interval": 7,
"created_on": None,
"contact": {"name": "Test User", "zipcode": "12345", "location": "Test City", "street": "", "phone": ""},
}
def remove_fields(config:dict[str, Any], *fields:str) -> dict[str, Any]:
"""Create a new ad configuration with specified fields removed."""
result = copy.deepcopy(config)
for field in fields:
if "." in field:
parts = field.split(".", maxsplit = 1)
current = result
for part in parts[:-1]:
if part in current:
current = current[part]
if parts[-1] in current:
del current[parts[-1]]
elif field in result:
del result[field]
return result
@pytest.fixture
def minimal_ad_config(base_ad_config:dict[str, Any]) -> dict[str, Any]:
"""Provide a minimal ad configuration with only required fields."""
return remove_fields(base_ad_config, "id", "created_on", "shipping_options", "special_attributes", "contact.street", "contact.phone")
class TestKleinanzeigenBotAdDeletion:
"""Tests for ad deletion functionality."""
@pytest.mark.asyncio
async def test_delete_ad_by_title_match_succeeds(
self,
test_bot:KleinanzeigenBot,
minimal_ad_config:dict[str, Any],
) -> None:
"""When title matches a published ad and server returns 200, should return True and clear id."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"title": "Test Title", "id": None})
published_ads = [{"title": "Test Title", "id": 67890}, {"title": "Other Title", "id": 11111}]
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock,
return_value = {"statusCode": 200, "statusMessage": "OK", "content": "{}"}),
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await delete_flow.delete_ad(
web = test_bot, root_url = test_bot.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = True,
)
assert result is True
assert ad_cfg.id is None
@pytest.mark.asyncio
async def test_delete_ad_by_id_succeeds(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When ad has an ID and server returns 200, should return True and clear id."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"id": 12345})
published_ads = [{"title": "Different Title", "id": 12345}, {"title": "Other Title", "id": 11111}]
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock,
return_value = {"statusCode": 200, "statusMessage": "OK", "content": "{}"}),
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await delete_flow.delete_ad(
web = test_bot, root_url = test_bot.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = False,
)
assert result is True
assert ad_cfg.id is None
@pytest.mark.asyncio
async def test_delete_ad_returns_false_when_no_match(
self,
test_bot:KleinanzeigenBot,
minimal_ad_config:dict[str, Any],
) -> None:
"""When no published ads match, should return False without opening any pages."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"title": "No Match Title", "id": 99999})
published_ads = [{"title": "Different Title", "id": 12345}]
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock) as mock_web_open,
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_web_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock) as mock_web_sleep,
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
):
result = await delete_flow.delete_ad(
web = test_bot, root_url = test_bot.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = True,
)
assert result is False
assert ad_cfg.id == 99999 # Preserved — no deletion attempted
mock_web_open.assert_not_called()
mock_web_find.assert_not_called()
mock_web_sleep.assert_not_called()
mock_request.assert_not_called()
@pytest.mark.asyncio
async def test_delete_ad_returns_false_on_404_clears_id(
self,
test_bot:KleinanzeigenBot,
minimal_ad_config:dict[str, Any],
) -> None:
"""When delete is attempted but server returns 404, should return False but still clear the id."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"id": 12345})
published_ads:list[dict[str, Any]] = []
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock, return_value = {"statusCode": 404, "statusMessage": "Not Found", "content": "{}"}),
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await delete_flow.delete_ad(
web = test_bot, root_url = test_bot.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = False,
)
assert result is False
assert ad_cfg.id is None # Cleared because delete was attempted
@pytest.mark.asyncio
async def test_delete_ad_skips_invalid_published_ad_id(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When a title-matched published ad has an invalid id (None), it should be skipped."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"title": "Test Title", "id": None})
published_ads:list[dict[str, Any]] = [
{"title": "Test Title", "id": None}, # Invalid — should be skipped
{"title": "Test Title", "id": "not-a-number"}, # Invalid — should be skipped
]
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock) as mock_web_open,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock),
):
result = await delete_flow.delete_ad(
web = test_bot, root_url = test_bot.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = True,
)
assert result is False
mock_web_open.assert_not_called() # No valid IDs → no page open
@pytest.mark.asyncio
async def test_delete_ad_with_zero_id(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When ad_cfg.id is 0 (falsy but valid), should still enter the ID-based deletion path."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"id": 0})
published_ads:list[dict[str, Any]] = []
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock,
return_value = {"statusCode": 200, "statusMessage": "OK", "content": "{}"}) as mock_request,
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await delete_flow.delete_ad(
web = test_bot, root_url = test_bot.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = False,
)
assert result is True
assert ad_cfg.id is None
mock_request.assert_called_once()
assert "ids=0" in mock_request.call_args[1]["url"]
@pytest.mark.asyncio
async def test_delete_ad_multiple_title_matches(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When multiple published ads match by title with mixed 200/404, should return True."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"title": "Test Title", "id": None})
published_ads = [
{"title": "Test Title", "id": 100},
{"title": "Test Title", "id": 200},
{"title": "Test Title", "id": 300},
]
response_sequence = [
{"statusCode": 200, "statusMessage": "OK", "content": "{}"},
{"statusCode": 404, "statusMessage": "Not Found", "content": "{}"},
{"statusCode": 200, "statusMessage": "OK", "content": "{}"},
]
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock, side_effect = response_sequence) as mock_request,
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await delete_flow.delete_ad(
web = test_bot, root_url = test_bot.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = True,
)
assert result is True
assert ad_cfg.id is None
assert mock_request.call_count == 3
@pytest.mark.asyncio
async def test_delete_ad_title_and_id_match_deduplicated(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When the same ad matches by both title and ID, it should be deleted only once."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"title": "Test Title", "id": 100})
published_ads = [{"title": "Test Title", "id": 100}]
ok_response = {"statusCode": 200, "statusMessage": "OK", "content": "{}"}
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock, return_value = ok_response) as mock_request,
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await delete_flow.delete_ad(
web = test_bot, root_url = test_bot.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = True,
)
assert result is True
assert ad_cfg.id is None
mock_request.assert_called_once() # Deduplicated — only one request
@pytest.mark.asyncio
async def test_delete_ad_exception_preserves_id(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When web_request raises an exception mid-loop, ad_cfg.id should be preserved and web_sleep not called."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"id": 12345})
published_ads:list[dict[str, Any]] = []
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock) as mock_web_sleep,
patch.object(test_bot, "web_request", new_callable = AsyncMock, side_effect = TimeoutError("request timed out")),
):
mock_find.return_value.attrs = {"content": "some-token"}
with pytest.raises(TimeoutError, match = "request timed out"):
await delete_flow.delete_ad(
web = test_bot, root_url = test_bot.root_url,
ad_cfg = ad_cfg,
published_ads_list = published_ads,
delete_old_ads_by_title = False,
)
assert ad_cfg.id == 12345 # Preserved — exception prevented clearing
mock_web_sleep.assert_not_called()
class TestDeleteAdsAfterDeletePolicy:
"""Tests for delete_ads orchestration with after_delete policy integration."""
@staticmethod
def _make_ad(minimal_ad_config:dict[str, Any], tmp_path:Path) -> tuple[str, Ad, dict[str, Any]]:
ad_cfg = Ad.model_validate(minimal_ad_config | {
"id": 12345, "active": True,
"created_on": "2024-06-01T12:00:00", "updated_on": "2024-06-10T08:30:00",
"content_hash": "abc123", "repost_count": 3, "price_reduction_count": 1,
})
return str(tmp_path / "ad.yaml"), ad_cfg, ad_cfg.model_dump()
@pytest.mark.asyncio
async def test_cleanup_on_404_detection(
self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any], tmp_path:Path,
) -> None:
"""Cleanup runs when delete_ad returns False but cleared the id (404 path)."""
test_bot.config.deleting.after_delete = "RESET"
ad_file, ad_cfg, ad_cfg_orig = self._make_ad(minimal_ad_config, tmp_path)
async def fake_delete(
_web:Any, _root_url:str, ad:Ad, _published:list[dict[str, Any]], **__:Any
) -> bool:
ad.id = None # Phase B ran and cleared the id
return False # but all responses were 404
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.delete_flow.delete_ad", new_callable = AsyncMock, side_effect = fake_delete),
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch("kleinanzeigen_bot.utils.dicts.save_dict") as mock_save,
):
await delete_flow.delete_ads(
web = test_bot, root_url = test_bot.root_url,
after_delete = test_bot.config.deleting.after_delete,
delete_old_ads_by_title = test_bot.config.publishing.delete_old_ads_by_title,
ad_cfgs = [(ad_file, ad_cfg, ad_cfg_orig)],
)
assert ad_cfg.repost_count == 0
assert "id" not in ad_cfg_orig
mock_save.assert_called_once()
@pytest.mark.asyncio
async def test_no_cleanup_when_delete_not_attempted(
self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any], tmp_path:Path,
) -> None:
"""No cleanup when delete_ad returns False with id preserved (no match)."""
test_bot.config.deleting.after_delete = "RESET"
ad_file, ad_cfg, ad_cfg_orig = self._make_ad(minimal_ad_config, tmp_path)
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.delete_flow.delete_ad", new_callable = AsyncMock, return_value = False),
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch("kleinanzeigen_bot.utils.dicts.save_dict") as mock_save,
):
await delete_flow.delete_ads(
web = test_bot, root_url = test_bot.root_url,
after_delete = test_bot.config.deleting.after_delete,
delete_old_ads_by_title = test_bot.config.publishing.delete_old_ads_by_title,
ad_cfgs = [(ad_file, ad_cfg, ad_cfg_orig)],
)
mock_save.assert_not_called()
assert ad_cfg.id == 12345
@pytest.mark.asyncio
async def test_delete_ads_counts_deletions(
self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any], tmp_path:Path,
) -> None:
"""Orchestrator increments deleted_count when delete_ad returns True."""
test_bot.config.deleting.after_delete = "NONE"
ad1 = self._make_ad(minimal_ad_config, tmp_path)
# Create second ad with different title/id
ad_cfg2 = Ad.model_validate(minimal_ad_config | {
"id": 67890, "title": "Second Ad Here", "active": True,
"created_on": "2024-06-01T12:00:00", "updated_on": "2024-06-10T08:30:00",
"content_hash": "def456",
})
ad2 = (str(tmp_path / "ad2.yaml"), ad_cfg2, ad_cfg2.model_dump())
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.delete_flow.delete_ad", new_callable = AsyncMock, return_value = True) as mock_delete_ad,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch("kleinanzeigen_bot.utils.dicts.save_dict") as mock_save,
):
await delete_flow.delete_ads(
web = test_bot, root_url = test_bot.root_url,
after_delete = test_bot.config.deleting.after_delete,
delete_old_ads_by_title = test_bot.config.publishing.delete_old_ads_by_title,
ad_cfgs = [ad1, ad2],
)
# save_dict not called because after_delete is NONE
mock_save.assert_not_called()
# delete_ad called twice (once per ad)
assert mock_delete_ad.call_count == 2

584
tests/unit/test_download_flow.py Normal file
View File

@@ -0,0 +1,584 @@
# SPDX-FileCopyrightText: © Jens Bergmann and contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
# SPDX-ArtifactOfProjectHomePage: https://github.com/Second-Hand-Friends/kleinanzeigen-bot/
"""Tests for download flow functionality."""
import logging
from pathlib import Path
from typing import Any, cast
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from kleinanzeigen_bot import KleinanzeigenBot, download_flow
from kleinanzeigen_bot.model.ad_model import Ad
from kleinanzeigen_bot.published_ads import PublishedAdsFetchIncompleteError
from kleinanzeigen_bot.utils import xdg_paths
class TestDownloadFlow:
"""Tests for download flow methods."""
@pytest.mark.asyncio
async def test_download_ads_passes_download_dir_and_published_ads(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
"""Ensure download_ads wires resolved download_dir and published_ads_by_id into AdExtractor."""
config_file = tmp_path / "config.yaml"
test_bot.workspace = xdg_paths.Workspace.for_config(config_file, "kleinanzeigen-bot")
test_bot.config_file_path = str(config_file)
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = [])
mock_published_ads = [{"id": 123, "buyNowEligible": True}, {"id": 456, "buyNowEligible": False}]
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = mock_published_ads),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock) as mock_extractor,
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
# Verify published_ads_by_id is built correctly and passed to extractor
mock_extractor.assert_called_once_with(
test_bot.browser,
test_bot.config,
test_bot.workspace.download_dir,
published_ads_by_id = {123: mock_published_ads[0], 456: mock_published_ads[1]},
)
def test_resolve_download_dir_uses_workspace_default_for_literal_default(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
config_file = tmp_path / "config.yaml"
test_bot.workspace = xdg_paths.Workspace.for_config(config_file, "kleinanzeigen-bot")
test_bot.config_file_path = str(config_file)
test_bot.browser = cast(Any, None)
assert download_flow.resolve_download_dir(test_bot.config, test_bot.config_file_path, test_bot.workspace) == test_bot.workspace.download_dir
def test_resolve_download_dir_uses_config_relative_path(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
config_file = tmp_path / "config.yaml"
test_bot.workspace = xdg_paths.Workspace.for_config(config_file, "kleinanzeigen-bot")
test_bot.config_file_path = str(config_file)
test_bot.browser = cast(Any, None)
test_bot.config.download.dir = "./my-ads"
assert download_flow.resolve_download_dir(test_bot.config, test_bot.config_file_path, test_bot.workspace) == (tmp_path / "my-ads").resolve()
def test_resolve_download_dir_uses_absolute_path(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
config_file = tmp_path / "config.yaml"
test_bot.workspace = xdg_paths.Workspace.for_config(config_file, "kleinanzeigen-bot")
test_bot.config_file_path = str(config_file)
test_bot.browser = cast(Any, None)
test_bot.config.download.dir = str((tmp_path / "absolute-target").resolve())
assert download_flow.resolve_download_dir(test_bot.config, test_bot.config_file_path, test_bot.workspace) == (tmp_path / "absolute-target").resolve()
@pytest.mark.asyncio
async def test_download_ads_uses_configured_relative_download_dir(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
config_file = tmp_path / "config.yaml"
test_bot.workspace = xdg_paths.Workspace.for_config(config_file, "kleinanzeigen-bot")
test_bot.config_file_path = str(config_file)
test_bot.config.download.dir = "./ads"
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = [])
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock) as mock_extractor,
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
mock_extractor.assert_called_once()
assert mock_extractor.call_args.args[2] == (tmp_path / "ads").resolve()
@pytest.mark.asyncio
@pytest.mark.parametrize(
"scenario",
[
{
"published_ads": [{"id": 123, "state": "active"}],
"expected_active": True,
},
{
"published_ads": [{"id": 999, "state": "active"}],
"expected_active": False,
},
{
"published_ads": [{"id": 123, "state": "inactive"}],
"expected_active": False,
},
{
"published_ads": [{"id": 123, "state": "paused"}],
"expected_active": False,
},
],
)
async def test_download_ads_numeric_selector_resolves_and_passes_active_state(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
scenario:dict[str, Any],
) -> None:
published_ads = scenario["published_ads"]
expected_active = scenario["expected_active"]
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "123"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = published_ads) as mock_fetch_published_ads,
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
mock_fetch_published_ads.assert_awaited_once_with(test_bot, test_bot.root_url, strict = True)
extractor_mock.download_ad.assert_awaited_once_with(123, active = expected_active)
@pytest.mark.asyncio
async def test_download_ads_numeric_selector_fails_when_published_ads_fetch_incomplete(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "123"
test_bot.browser = MagicMock()
with (
patch(
"kleinanzeigen_bot.published_ads.fetch_published_ads",
new_callable = AsyncMock,
side_effect = PublishedAdsFetchIncompleteError("incomplete fetch"),
),
patch("kleinanzeigen_bot.extract.AdExtractor") as mock_extractor,
pytest.raises(PublishedAdsFetchIncompleteError, match = "incomplete fetch"),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
mock_extractor.assert_not_called()
@pytest.mark.asyncio
async def test_download_ads_all_selector_uses_tolerant_published_ads_fetch(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = [])
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = []) as mock_fetch_published_ads,
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
mock_fetch_published_ads.assert_awaited_once_with(test_bot, test_bot.root_url, strict = False)
@pytest.mark.asyncio
@pytest.mark.parametrize(
"scenario",
[
{
"name": "active ad",
"published_ads": [{"id": 123, "state": "active"}],
"expected_active": True,
"expect_ownership_warning": False,
},
{
"name": "inactive ad",
"published_ads": [{"id": 123, "state": "inactive"}],
"expected_active": False,
"expect_ownership_warning": False,
},
{
"name": "paused ad",
"published_ads": [{"id": 123, "state": "paused"}],
"expected_active": False,
"expect_ownership_warning": False,
},
{
"name": "ad not in published profile",
"published_ads": [{"id": 999, "state": "active"}], # Different ID
"expected_active": False,
"expect_ownership_warning": True,
},
],
ids = lambda s: s["name"],
)
async def test_download_ads_all_selector_resolves_and_passes_active_state(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
caplog:pytest.LogCaptureFixture,
scenario:dict[str, Any],
) -> None:
"""Test that --ads=all resolves and passes correct active state to download_ad."""
# Setup
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test-ad/123-234-5678"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 123)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
caplog.set_level(logging.WARNING)
with (
patch(
"kleinanzeigen_bot.published_ads.fetch_published_ads",
new_callable = AsyncMock,
return_value = scenario["published_ads"],
) as mock_fetch_published_ads,
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
# Verify published ads fetched with strict=False (tolerant mode for "all")
mock_fetch_published_ads.assert_awaited_once_with(test_bot, test_bot.root_url, strict = False)
# Verify download_ad called with correct active parameter
extractor_mock.download_ad.assert_awaited_once_with(123, active = scenario["expected_active"])
# Verify ownership warning only when expected
ownership_warnings = [msg for msg in caplog.messages if "found in overview but not in published profile" in msg]
if scenario["expect_ownership_warning"]:
assert len(ownership_warnings) == 1
else:
assert len(ownership_warnings) == 0
@pytest.mark.asyncio
async def test_download_ads_all_selector_skips_invalid_ad_id(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
) -> None:
"""Test that --ads=all skips ads with invalid URL parsing (ad_id=-1)."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test/invalid-url"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = -1) # URL parsing failed
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
# Verify download_ad was NOT called for invalid ad_id
extractor_mock.download_ad.assert_not_called()
@pytest.mark.asyncio
@pytest.mark.parametrize(
"scenario",
[
{
"name": "new active ad",
"published_ads": [{"id": 999, "state": "active"}],
"saved_ad_ids": [123, 456], # 999 is not saved, so it's "new"
"expected_active": True,
},
{
"name": "new inactive ad",
"published_ads": [{"id": 999, "state": "inactive"}],
"saved_ad_ids": [123, 456], # 999 is not saved, so it's "new"
"expected_active": False,
},
{
"name": "new paused ad",
"published_ads": [{"id": 999, "state": "paused"}],
"saved_ad_ids": [123, 456], # 999 is not saved, so it's "new"
"expected_active": False,
},
],
ids = lambda s: s["name"],
)
async def test_download_ads_new_selector_resolves_and_passes_active_state(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
scenario:dict[str, Any],
) -> None:
"""Test that --ads=new resolves and passes correct active state to download_ad."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "new"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test-ad/999-234-5678"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 999)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
# Mock load_ads to return the saved_ad_ids
saved_ads:list[tuple[str, MagicMock, dict[str, Any]]] = [
(
f"ad_{ad_id}.yaml",
MagicMock(spec = Ad, id = ad_id),
{},
)
for ad_id in scenario["saved_ad_ids"]
]
with (
patch(
"kleinanzeigen_bot.published_ads.fetch_published_ads",
new_callable = AsyncMock,
return_value = scenario["published_ads"],
) as mock_fetch_published_ads,
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
patch.object(test_bot, "load_ads", return_value = saved_ads),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
# Verify published ads fetched with strict=False (tolerant mode for "new")
mock_fetch_published_ads.assert_awaited_once_with(test_bot, test_bot.root_url, strict = False)
# Verify download_ad called with correct active parameter
extractor_mock.download_ad.assert_awaited_once_with(999, active = scenario["expected_active"])
@pytest.mark.asyncio
async def test_download_ads_new_selector_skips_already_saved(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
) -> None:
"""Test that --ads=new skips already-saved ads (existing behavior unchanged)."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "new"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test-ad/123-234-5678"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 123)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
# Mock load_ads to return ad 123 as already saved
saved_ads:list[tuple[str, MagicMock, dict[str, Any]]] = [("ad_123.yaml", MagicMock(spec = Ad, id = 123), {})]
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
patch.object(test_bot, "load_ads", return_value = saved_ads),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
# Verify download_ad was NOT called for already-saved ad
extractor_mock.download_ad.assert_not_called()
@pytest.mark.asyncio
async def test_download_ads_new_selector_skips_invalid_ad_id(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
) -> None:
"""Test that --ads=new skips ads with invalid URL parsing (ad_id=-1)."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "new"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test/invalid-url"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = -1) # URL parsing failed
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
patch.object(test_bot, "load_ads", return_value = []),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
# Verify download_ad was NOT called for invalid ad_id
extractor_mock.download_ad.assert_not_called()
@pytest.mark.asyncio
async def test_download_ads_new_selector_passes_inactive_for_ad_not_in_published_profile(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
) -> None:
"""Test that --ads=new passes active=False when ad is not in published profile."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "new"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test-ad/999-234-5678"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 999)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
# Mock load_ads to return different saved ads (999 is new but not in published profile)
saved_ads:list[tuple[str, MagicMock, dict[str, Any]]] = [("ad_123.yaml", MagicMock(spec = Ad, id = 123), {})]
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
patch.object(test_bot, "load_ads", return_value = saved_ads),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
# Verify download_ad was called with active=False (not in profile)
extractor_mock.download_ad.assert_awaited_once_with(999, active = False)
@pytest.mark.asyncio
async def test_download_ads_all_selector_skips_when_navigation_fails(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
) -> None:
"""Test that --ads=all skips download when navigate_to_ad_page returns False."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test/123"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 123)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = False) # Navigation fails
extractor_mock.download_ad = AsyncMock()
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", AsyncMock(return_value = [{"id": 123, "state": "active"}])),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
# Verify download_ad was NOT called when navigation fails
extractor_mock.download_ad.assert_not_called()
@pytest.mark.asyncio
@pytest.mark.parametrize("state_value", ["", "deleted", "expired", "pending", "draft", None])
async def test_download_ads_all_selector_treats_unexpected_states_as_inactive(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
state_value:str | None,
) -> None:
"""Test that unexpected state values are treated as inactive."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
published_ads = [{"id": 123, "state": state_value}] if state_value is not None else [{"id": 123}]
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test/123"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 123)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
with (
patch("kleinanzeigen_bot.published_ads.fetch_published_ads", AsyncMock(return_value = published_ads)),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await download_flow.download_ads(
web = test_bot, config = test_bot.config,
config_file_path = test_bot.config_file_path,
workspace = test_bot.workspace,
ads_selector = test_bot.ads_selector,
load_ads_func = test_bot.load_ads,
root_url = test_bot.root_url,
)
# All non-"active" states should result in active=False
extractor_mock.download_ad.assert_awaited_once_with(123, active = False)

128
tests/unit/test_extend_command.py → tests/unit/test_extend_flow.py
View File

@@ -1,7 +1,9 @@
# SPDX-FileCopyrightText: © Jens Bergmann and contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
# SPDX-ArtifactOfProjectHomePage: https://github.com/Second-Hand-Friends/kleinanzeigen-bot/
import json # isort: skip
"""Tests for the extend command and extend_flow module."""
import json
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any
@@ -9,7 +11,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from kleinanzeigen_bot import KleinanzeigenBot, runtime_config
from kleinanzeigen_bot import KleinanzeigenBot, extend_flow, runtime_config
from kleinanzeigen_bot.model.ad_model import Ad
from kleinanzeigen_bot.utils import dicts, misc, xdg_paths
from kleinanzeigen_bot.utils.web_scraping_mixin import By, Element
@@ -99,7 +101,7 @@ class TestExtendAdsMethod:
with patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request, patch.object(test_bot, "web_sleep", new_callable = AsyncMock):
mock_request.return_value = {"content": '{"ads": []}'}
await test_bot.extend_ads([("test.yaml", ad_cfg, ad_config)])
await extend_flow.extend_ads(web = test_bot, root_url = test_bot.root_url, ad_cfgs = [("test.yaml", ad_cfg, ad_config)])
# Verify no extension was attempted
mock_request.assert_called_once() # Only the API call to get published ads
@@ -113,7 +115,7 @@ class TestExtendAdsMethod:
# Return empty published ads list
mock_request.return_value = {"content": '{"ads": []}'}
await test_bot.extend_ads([("test.yaml", ad_cfg, base_ad_config_with_id)])
await extend_flow.extend_ads(web = test_bot, root_url = test_bot.root_url, ad_cfgs = [("test.yaml", ad_cfg, base_ad_config_with_id)])
# Verify no extension was attempted
mock_request.assert_called_once()
@@ -137,11 +139,11 @@ class TestExtendAdsMethod:
with (
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "extend_ad", new_callable = AsyncMock) as mock_extend_ad,
patch("kleinanzeigen_bot.extend_flow._extend_ad", new_callable = AsyncMock) as mock_extend_ad,
):
mock_request.return_value = {"content": json.dumps(published_ads_json)}
await test_bot.extend_ads([("test.yaml", ad_cfg, base_ad_config_with_id)])
await extend_flow.extend_ads(web = test_bot, root_url = test_bot.root_url, ad_cfgs = [("test.yaml", ad_cfg, base_ad_config_with_id)])
# Verify extend_ad was not called
mock_extend_ad.assert_not_called()
@@ -165,11 +167,11 @@ class TestExtendAdsMethod:
with (
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "extend_ad", new_callable = AsyncMock) as mock_extend_ad,
patch("kleinanzeigen_bot.extend_flow._extend_ad", new_callable = AsyncMock) as mock_extend_ad,
):
mock_request.return_value = {"content": json.dumps(published_ads_json)}
await test_bot.extend_ads([("test.yaml", ad_cfg, base_ad_config_with_id)])
await extend_flow.extend_ads(web = test_bot, root_url = test_bot.root_url, ad_cfgs = [("test.yaml", ad_cfg, base_ad_config_with_id)])
# Verify extend_ad was not called
mock_extend_ad.assert_not_called()
@@ -188,11 +190,11 @@ class TestExtendAdsMethod:
with (
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "extend_ad", new_callable = AsyncMock) as mock_extend_ad,
patch("kleinanzeigen_bot.extend_flow._extend_ad", new_callable = AsyncMock) as mock_extend_ad,
):
mock_request.return_value = {"content": json.dumps(published_ads_json)}
await test_bot.extend_ads([("test.yaml", ad_cfg, base_ad_config_with_id)])
await extend_flow.extend_ads(web = test_bot, root_url = test_bot.root_url, ad_cfgs = [("test.yaml", ad_cfg, base_ad_config_with_id)])
# Verify extend_ad was not called
mock_extend_ad.assert_not_called()
@@ -211,12 +213,12 @@ class TestExtendAdsMethod:
with (
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "extend_ad", new_callable = AsyncMock) as mock_extend_ad,
patch("kleinanzeigen_bot.extend_flow._extend_ad", new_callable = AsyncMock) as mock_extend_ad,
):
mock_request.return_value = {"content": json.dumps(published_ads_json)}
mock_extend_ad.return_value = True
await test_bot.extend_ads([("test.yaml", ad_cfg, base_ad_config_with_id)])
await extend_flow.extend_ads(web = test_bot, root_url = test_bot.root_url, ad_cfgs = [("test.yaml", ad_cfg, base_ad_config_with_id)])
# Verify extend_ad was called
mock_extend_ad.assert_called_once()
@@ -246,24 +248,56 @@ class TestExtendAdsMethod:
with (
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "extend_ad", new_callable = AsyncMock) as mock_extend_ad,
patch("kleinanzeigen_bot.extend_flow._extend_ad", new_callable = AsyncMock) as mock_extend_ad,
):
mock_request.return_value = {"content": json.dumps(published_ads_json)}
mock_extend_ad.return_value = True
await test_bot.extend_ads([("test1.yaml", ad_cfg1, base_ad_config_with_id), ("test2.yaml", ad_cfg2, ad_config2)])
await extend_flow.extend_ads(
web = test_bot, root_url = test_bot.root_url,
ad_cfgs = [("test1.yaml", ad_cfg1, base_ad_config_with_id), ("test2.yaml", ad_cfg2, ad_config2)],
)
# Verify extend_ad was called only once (for the ad within window)
assert mock_extend_ad.call_count == 1
@pytest.mark.asyncio
async def test_extend_ads_skips_ad_with_invalid_enddate(self, test_bot:KleinanzeigenBot, base_ad_config_with_id:dict[str, Any]) -> None:
"""Test that extend_ads gracefully skips ads with invalid endDate format instead of crashing."""
ad_cfg = Ad.model_validate(base_ad_config_with_id)
published_ads_json = {
"ads": [
{
"id": 12345,
"title": "Test Ad Title",
"state": "active",
"endDate": "not-a-date", # Invalid format
}
]
}
with (
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch("kleinanzeigen_bot.extend_flow._extend_ad", new_callable = AsyncMock) as mock_extend_ad,
):
mock_request.return_value = {"content": json.dumps(published_ads_json)}
# Should not raise — gracefully skips invalid endDate
await extend_flow.extend_ads(web = test_bot, root_url = test_bot.root_url, ad_cfgs = [("test.yaml", ad_cfg, base_ad_config_with_id)])
# Verify extend_ad was NOT called (invalid endDate → skip)
mock_extend_ad.assert_not_called()
class TestExtendAdMethod:
"""Tests for the extend_ad() method.
"""Tests for the _extend_ad() method.
Note: These tests mock `_navigate_paginated_ad_overview` rather than individual browser methods
Note: These tests mock `navigate_paginated_ad_overview` rather than individual browser methods
(web_find, web_click, etc.) because the pagination helper involves complex multi-step browser
interactions that would require extensive, brittle mock choreography. Mocking at this level
keeps tests focused on extend_ad's own logic (dialog handling, YAML persistence, error paths).
keeps tests focused on _extend_ad's own logic (dialog handling, YAML persistence, error paths).
"""
@pytest.mark.asyncio
@@ -276,7 +310,7 @@ class TestExtendAdMethod:
dicts.save_dict(str(ad_file), base_ad_config_with_id)
with (
patch.object(test_bot, "_navigate_paginated_ad_overview", new_callable = AsyncMock) as mock_paginate,
patch.object(test_bot, "navigate_paginated_ad_overview", new_callable = AsyncMock) as mock_paginate,
patch.object(test_bot, "web_click", new_callable = AsyncMock),
patch("kleinanzeigen_bot.utils.misc.now") as mock_now,
):
@@ -285,7 +319,11 @@ class TestExtendAdMethod:
mock_paginate.return_value = True
result = await test_bot.extend_ad(str(ad_file), ad_cfg, base_ad_config_with_id)
result = await extend_flow._extend_ad(
web = test_bot, root_url = test_bot.root_url,
ad_file = str(ad_file), ad_cfg = ad_cfg,
ad_cfg_orig = base_ad_config_with_id,
)
assert result is True
assert mock_paginate.call_count == 1
@@ -296,25 +334,29 @@ class TestExtendAdMethod:
@pytest.mark.asyncio
async def test_extend_ad_button_not_found(self, test_bot:KleinanzeigenBot, base_ad_config_with_id:dict[str, Any], tmp_path:Path) -> None:
"""Test extend_ad when the Verlängern button is not found."""
"""Test _extend_ad when the Verlängern button is not found."""
ad_cfg = Ad.model_validate(base_ad_config_with_id)
# Create temporary YAML file
ad_file = tmp_path / "test_ad.yaml"
dicts.save_dict(str(ad_file), base_ad_config_with_id)
with patch.object(test_bot, "_navigate_paginated_ad_overview", new_callable = AsyncMock) as mock_paginate:
with patch.object(test_bot, "navigate_paginated_ad_overview", new_callable = AsyncMock) as mock_paginate:
# Simulate button not found by having pagination return False (not found on any page)
mock_paginate.return_value = False
result = await test_bot.extend_ad(str(ad_file), ad_cfg, base_ad_config_with_id)
result = await extend_flow._extend_ad(
web = test_bot, root_url = test_bot.root_url,
ad_file = str(ad_file), ad_cfg = ad_cfg,
ad_cfg_orig = base_ad_config_with_id,
)
assert result is False
assert mock_paginate.call_count == 1
@pytest.mark.asyncio
async def test_extend_ad_dialog_timeout(self, test_bot:KleinanzeigenBot, base_ad_config_with_id:dict[str, Any], tmp_path:Path) -> None:
"""Test extend_ad when the confirmation dialog times out (no dialog appears)."""
"""Test _extend_ad when the confirmation dialog times out (no dialog appears)."""
ad_cfg = Ad.model_validate(base_ad_config_with_id)
# Create temporary YAML file
@@ -322,7 +364,7 @@ class TestExtendAdMethod:
dicts.save_dict(str(ad_file), base_ad_config_with_id)
with (
patch.object(test_bot, "_navigate_paginated_ad_overview", new_callable = AsyncMock) as mock_paginate,
patch.object(test_bot, "navigate_paginated_ad_overview", new_callable = AsyncMock) as mock_paginate,
patch.object(test_bot, "web_click", new_callable = AsyncMock) as mock_click,
patch("kleinanzeigen_bot.utils.misc.now") as mock_now,
):
@@ -334,30 +376,38 @@ class TestExtendAdMethod:
# Dialog close button times out
mock_click.side_effect = TimeoutError("Dialog not found")
result = await test_bot.extend_ad(str(ad_file), ad_cfg, base_ad_config_with_id)
result = await extend_flow._extend_ad(
web = test_bot, root_url = test_bot.root_url,
ad_file = str(ad_file), ad_cfg = ad_cfg,
ad_cfg_orig = base_ad_config_with_id,
)
# Should still succeed (dialog might not appear)
assert result is True
@pytest.mark.asyncio
async def test_extend_ad_exception_handling(self, test_bot:KleinanzeigenBot, base_ad_config_with_id:dict[str, Any], tmp_path:Path) -> None:
"""Test extend_ad propagates unexpected exceptions."""
"""Test _extend_ad propagates unexpected exceptions."""
ad_cfg = Ad.model_validate(base_ad_config_with_id)
# Create temporary YAML file
ad_file = tmp_path / "test_ad.yaml"
dicts.save_dict(str(ad_file), base_ad_config_with_id)
with patch.object(test_bot, "_navigate_paginated_ad_overview", new_callable = AsyncMock) as mock_paginate:
with patch.object(test_bot, "navigate_paginated_ad_overview", new_callable = AsyncMock) as mock_paginate:
# Simulate unexpected exception during pagination
mock_paginate.side_effect = Exception("Unexpected error")
with pytest.raises(Exception, match = "Unexpected error"):
await test_bot.extend_ad(str(ad_file), ad_cfg, base_ad_config_with_id)
await extend_flow._extend_ad(
web = test_bot, root_url = test_bot.root_url,
ad_file = str(ad_file), ad_cfg = ad_cfg,
ad_cfg_orig = base_ad_config_with_id,
)
@pytest.mark.asyncio
async def test_extend_ad_with_web_mocks(self, test_bot:KleinanzeigenBot, base_ad_config_with_id:dict[str, Any], tmp_path:Path) -> None:
"""Test extend_ad with web-level mocks to exercise the find_and_click_extend_button callback."""
"""Test _extend_ad with web-level mocks to exercise the find_and_click_extend_button callback."""
ad_cfg = Ad.model_validate(base_ad_config_with_id)
# Create temporary YAML file
@@ -394,13 +444,17 @@ class TestExtendAdMethod:
patch.object(test_bot, "web_find_all", new_callable = AsyncMock, return_value = []),
patch.object(test_bot, "web_scroll_page_down", new_callable = AsyncMock),
patch.object(test_bot, "web_click", new_callable = AsyncMock),
patch.object(test_bot, "_timeout", return_value = 10),
patch.object(test_bot, "timeout", return_value = 10),
patch("kleinanzeigen_bot.utils.misc.now") as mock_now,
):
# Test mock datetime - timezone not relevant for timestamp formatting test
mock_now.return_value = datetime(2025, 1, 28, 15, 0, 0) # noqa: DTZ001
result = await test_bot.extend_ad(str(ad_file), ad_cfg, base_ad_config_with_id)
result = await extend_flow._extend_ad(
web = test_bot, root_url = test_bot.root_url,
ad_file = str(ad_file), ad_cfg = ad_cfg,
ad_cfg_orig = base_ad_config_with_id,
)
assert result is True
# Verify the extend button was found and clicked
@@ -428,12 +482,12 @@ class TestExtendEdgeCases:
with (
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "extend_ad", new_callable = AsyncMock) as mock_extend_ad,
patch("kleinanzeigen_bot.extend_flow._extend_ad", new_callable = AsyncMock) as mock_extend_ad,
):
mock_request.return_value = {"content": json.dumps(published_ads_json)}
mock_extend_ad.return_value = True
await test_bot.extend_ads([("test.yaml", ad_cfg, base_ad_config_with_id)])
await extend_flow.extend_ads(web = test_bot, root_url = test_bot.root_url, ad_cfgs = [("test.yaml", ad_cfg, base_ad_config_with_id)])
# Verify extend_ad was called (8 days is within the window)
mock_extend_ad.assert_called_once()
@@ -452,11 +506,11 @@ class TestExtendEdgeCases:
with (
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "extend_ad", new_callable = AsyncMock) as mock_extend_ad,
patch("kleinanzeigen_bot.extend_flow._extend_ad", new_callable = AsyncMock) as mock_extend_ad,
):
mock_request.return_value = {"content": json.dumps(published_ads_json)}
await test_bot.extend_ads([("test.yaml", ad_cfg, base_ad_config_with_id)])
await extend_flow.extend_ads(web = test_bot, root_url = test_bot.root_url, ad_cfgs = [("test.yaml", ad_cfg, base_ad_config_with_id)])
# Verify extend_ad was not called (9 days is outside the window)
mock_extend_ad.assert_not_called()
@@ -481,7 +535,7 @@ class TestExtendEdgeCases:
with (
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "extend_ad", new_callable = AsyncMock) as mock_extend_ad,
patch("kleinanzeigen_bot.extend_flow._extend_ad", new_callable = AsyncMock) as mock_extend_ad,
patch("kleinanzeigen_bot.utils.misc.now") as mock_now,
):
# Mock now() to return a date where 05.02.2026 would be within 8 days
@@ -490,7 +544,7 @@ class TestExtendEdgeCases:
mock_request.return_value = {"content": json.dumps(published_ads_json)}
mock_extend_ad.return_value = True
await test_bot.extend_ads([("test.yaml", ad_cfg, base_ad_config_with_id)])
await extend_flow.extend_ads(web = test_bot, root_url = test_bot.root_url, ad_cfgs = [("test.yaml", ad_cfg, base_ad_config_with_id)])
# Verify extend_ad was called (date was parsed correctly)
mock_extend_ad.assert_called_once()

767
tests/unit/test_init.py
View File

@@ -28,7 +28,7 @@ from kleinanzeigen_bot.model.config_model import (
PublishingConfig,
)
from kleinanzeigen_bot.utils import xdg_paths
from kleinanzeigen_bot.utils.exceptions import CategoryResolutionError, PublishedAdsFetchIncompleteError, PublishSubmissionUncertainError
from kleinanzeigen_bot.utils.exceptions import CategoryResolutionError, PublishSubmissionUncertainError
from kleinanzeigen_bot.utils.web_scraping_mixin import By, Element
@@ -159,7 +159,9 @@ class TestKleinanzeigenBotInitialization:
patch.object(test_bot, "load_ads", return_value = []),
patch.object(test_bot, "create_browser_session", new_callable = AsyncMock),
patch.object(test_bot, "login", new_callable = AsyncMock),
patch.object(test_bot, "download_ads", new_callable = AsyncMock),
patch("kleinanzeigen_bot.download_flow.download_ads", new_callable = AsyncMock),
patch.object(test_bot, "close_browser_session"),
patch("kleinanzeigen_bot.UpdateChecker", DummyUpdateChecker),
):
@@ -168,472 +170,6 @@ class TestKleinanzeigenBotInitialization:
expected_state_path = (tmp_path / "config.yaml").resolve().parent / ".temp" / "update_check_state.json"
assert update_checker_calls == [(test_bot.config, expected_state_path)]
@pytest.mark.asyncio
async def test_download_ads_passes_download_dir_and_published_ads(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
"""Ensure download_ads wires resolved download_dir and published_ads_by_id into AdExtractor."""
config_file = tmp_path / "config.yaml"
test_bot.workspace = xdg_paths.Workspace.for_config(config_file, "kleinanzeigen-bot")
test_bot.config_file_path = str(config_file)
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = [])
mock_published_ads = [{"id": 123, "buyNowEligible": True}, {"id": 456, "buyNowEligible": False}]
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = mock_published_ads),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock) as mock_extractor,
):
await test_bot.download_ads()
# Verify published_ads_by_id is built correctly and passed to extractor
mock_extractor.assert_called_once_with(
test_bot.browser,
test_bot.config,
test_bot.workspace.download_dir,
published_ads_by_id = {123: mock_published_ads[0], 456: mock_published_ads[1]},
)
def test_resolve_download_dir_uses_workspace_default_for_literal_default(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
config_file = tmp_path / "config.yaml"
test_bot.workspace = xdg_paths.Workspace.for_config(config_file, "kleinanzeigen-bot")
test_bot.config_file_path = str(config_file)
test_bot.browser = cast(Any, None)
assert test_bot._resolve_download_dir() == test_bot.workspace.download_dir
def test_resolve_download_dir_uses_config_relative_path(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
config_file = tmp_path / "config.yaml"
test_bot.workspace = xdg_paths.Workspace.for_config(config_file, "kleinanzeigen-bot")
test_bot.config_file_path = str(config_file)
test_bot.browser = cast(Any, None)
test_bot.config.download.dir = "./my-ads"
assert test_bot._resolve_download_dir() == (tmp_path / "my-ads").resolve()
def test_resolve_download_dir_uses_absolute_path(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
config_file = tmp_path / "config.yaml"
test_bot.workspace = xdg_paths.Workspace.for_config(config_file, "kleinanzeigen-bot")
test_bot.config_file_path = str(config_file)
test_bot.browser = cast(Any, None)
test_bot.config.download.dir = str((tmp_path / "absolute-target").resolve())
assert test_bot._resolve_download_dir() == (tmp_path / "absolute-target").resolve()
@pytest.mark.asyncio
async def test_download_ads_uses_configured_relative_download_dir(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
config_file = tmp_path / "config.yaml"
test_bot.workspace = xdg_paths.Workspace.for_config(config_file, "kleinanzeigen-bot")
test_bot.config_file_path = str(config_file)
test_bot.config.download.dir = "./ads"
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = [])
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock) as mock_extractor,
):
await test_bot.download_ads()
mock_extractor.assert_called_once()
assert mock_extractor.call_args.args[2] == (tmp_path / "ads").resolve()
@pytest.mark.asyncio
@pytest.mark.parametrize(
"scenario",
[
{
"published_ads": [{"id": 123, "state": "active"}],
"expected_active": True,
},
{
"published_ads": [{"id": 999, "state": "active"}],
"expected_active": False,
},
{
"published_ads": [{"id": 123, "state": "inactive"}],
"expected_active": False,
},
{
"published_ads": [{"id": 123, "state": "paused"}],
"expected_active": False,
},
],
)
async def test_download_ads_numeric_selector_resolves_and_passes_active_state(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
scenario:dict[str, Any],
) -> None:
published_ads = scenario["published_ads"]
expected_active = scenario["expected_active"]
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "123"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = published_ads) as mock_fetch_published_ads,
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await test_bot.download_ads()
mock_fetch_published_ads.assert_awaited_once_with(strict = True)
extractor_mock.download_ad.assert_awaited_once_with(123, active = expected_active)
@pytest.mark.asyncio
async def test_download_ads_numeric_selector_fails_when_published_ads_fetch_incomplete(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "123"
test_bot.browser = MagicMock()
with (
patch.object(
test_bot,
"_fetch_published_ads",
new_callable = AsyncMock,
side_effect = PublishedAdsFetchIncompleteError("incomplete fetch"),
),
patch("kleinanzeigen_bot.extract.AdExtractor") as mock_extractor,
pytest.raises(PublishedAdsFetchIncompleteError, match = "incomplete fetch"),
):
await test_bot.download_ads()
mock_extractor.assert_not_called()
@pytest.mark.asyncio
async def test_download_ads_all_selector_uses_tolerant_published_ads_fetch(self, test_bot:KleinanzeigenBot, tmp_path:Path) -> None:
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = [])
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = []) as mock_fetch_published_ads,
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await test_bot.download_ads()
mock_fetch_published_ads.assert_awaited_once_with(strict = False)
@pytest.mark.asyncio
@pytest.mark.parametrize(
"scenario",
[
{
"name": "active ad",
"published_ads": [{"id": 123, "state": "active"}],
"expected_active": True,
"expect_ownership_warning": False,
},
{
"name": "inactive ad",
"published_ads": [{"id": 123, "state": "inactive"}],
"expected_active": False,
"expect_ownership_warning": False,
},
{
"name": "paused ad",
"published_ads": [{"id": 123, "state": "paused"}],
"expected_active": False,
"expect_ownership_warning": False,
},
{
"name": "ad not in published profile",
"published_ads": [{"id": 999, "state": "active"}], # Different ID
"expected_active": False,
"expect_ownership_warning": True,
},
],
ids = lambda s: s["name"],
)
async def test_download_ads_all_selector_resolves_and_passes_active_state(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
caplog:pytest.LogCaptureFixture,
scenario:dict[str, Any],
) -> None:
"""Test that --ads=all resolves and passes correct active state to download_ad."""
# Setup
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test-ad/123-234-5678"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 123)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
caplog.set_level(logging.WARNING)
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = scenario["published_ads"]) as mock_fetch_published_ads,
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await test_bot.download_ads()
# Verify published ads fetched with strict=False (tolerant mode for "all")
mock_fetch_published_ads.assert_awaited_once_with(strict = False)
# Verify download_ad called with correct active parameter
extractor_mock.download_ad.assert_awaited_once_with(123, active = scenario["expected_active"])
# Verify ownership warning only when expected
ownership_warnings = [msg for msg in caplog.messages if "found in overview but not in published profile" in msg]
if scenario["expect_ownership_warning"]:
assert len(ownership_warnings) == 1
else:
assert len(ownership_warnings) == 0
@pytest.mark.asyncio
async def test_download_ads_all_selector_skips_invalid_ad_id(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
) -> None:
"""Test that --ads=all skips ads with invalid URL parsing (ad_id=-1)."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test/invalid-url"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = -1) # URL parsing failed
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await test_bot.download_ads()
# Verify download_ad was NOT called for invalid ad_id
extractor_mock.download_ad.assert_not_called()
@pytest.mark.asyncio
@pytest.mark.parametrize(
"scenario",
[
{
"name": "new active ad",
"published_ads": [{"id": 999, "state": "active"}],
"saved_ad_ids": [123, 456], # 999 is not saved, so it's "new"
"expected_active": True,
},
{
"name": "new inactive ad",
"published_ads": [{"id": 999, "state": "inactive"}],
"saved_ad_ids": [123, 456], # 999 is not saved, so it's "new"
"expected_active": False,
},
{
"name": "new paused ad",
"published_ads": [{"id": 999, "state": "paused"}],
"saved_ad_ids": [123, 456], # 999 is not saved, so it's "new"
"expected_active": False,
},
],
ids = lambda s: s["name"],
)
async def test_download_ads_new_selector_resolves_and_passes_active_state(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
scenario:dict[str, Any],
) -> None:
"""Test that --ads=new resolves and passes correct active state to download_ad."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "new"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test-ad/999-234-5678"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 999)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
# Mock load_ads to return the saved_ad_ids
saved_ads:list[tuple[str, MagicMock, dict[str, Any]]] = [
(
f"ad_{ad_id}.yaml",
MagicMock(spec = Ad, id = ad_id),
{},
)
for ad_id in scenario["saved_ad_ids"]
]
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = scenario["published_ads"]) as mock_fetch_published_ads,
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
patch.object(test_bot, "load_ads", return_value = saved_ads),
):
await test_bot.download_ads()
# Verify published ads fetched with strict=False (tolerant mode for "new")
mock_fetch_published_ads.assert_awaited_once_with(strict = False)
# Verify download_ad called with correct active parameter
extractor_mock.download_ad.assert_awaited_once_with(999, active = scenario["expected_active"])
@pytest.mark.asyncio
async def test_download_ads_new_selector_skips_already_saved(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
) -> None:
"""Test that --ads=new skips already-saved ads (existing behavior unchanged)."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "new"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test-ad/123-234-5678"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 123)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
# Mock load_ads to return ad 123 as already saved
saved_ads:list[tuple[str, MagicMock, dict[str, Any]]] = [("ad_123.yaml", MagicMock(spec = Ad, id = 123), {})]
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
patch.object(test_bot, "load_ads", return_value = saved_ads),
):
await test_bot.download_ads()
# Verify download_ad was NOT called for already-saved ad
extractor_mock.download_ad.assert_not_called()
@pytest.mark.asyncio
async def test_download_ads_new_selector_skips_invalid_ad_id(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
) -> None:
"""Test that --ads=new skips ads with invalid URL parsing (ad_id=-1)."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "new"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test/invalid-url"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = -1) # URL parsing failed
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
patch.object(test_bot, "load_ads", return_value = []),
):
await test_bot.download_ads()
# Verify download_ad was NOT called for invalid ad_id
extractor_mock.download_ad.assert_not_called()
@pytest.mark.asyncio
async def test_download_ads_new_selector_passes_inactive_for_ad_not_in_published_profile(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
) -> None:
"""Test that --ads=new passes active=False when ad is not in published profile."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "new"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test-ad/999-234-5678"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 999)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
# Mock load_ads to return different saved ads (999 is new but not in published profile)
saved_ads:list[tuple[str, MagicMock, dict[str, Any]]] = [("ad_123.yaml", MagicMock(spec = Ad, id = 123), {})]
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
patch.object(test_bot, "load_ads", return_value = saved_ads),
):
await test_bot.download_ads()
# Verify download_ad was called with active=False (not in profile)
extractor_mock.download_ad.assert_awaited_once_with(999, active = False)
@pytest.mark.asyncio
async def test_download_ads_all_selector_skips_when_navigation_fails(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
) -> None:
"""Test that --ads=all skips download when navigate_to_ad_page returns False."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test/123"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 123)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = False) # Navigation fails
extractor_mock.download_ad = AsyncMock()
with (
patch.object(test_bot, "_fetch_published_ads", AsyncMock(return_value = [{"id": 123, "state": "active"}])),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await test_bot.download_ads()
# Verify download_ad was NOT called when navigation fails
extractor_mock.download_ad.assert_not_called()
@pytest.mark.asyncio
@pytest.mark.parametrize("state_value", ["", "deleted", "expired", "pending", "draft", None])
async def test_download_ads_all_selector_treats_unexpected_states_as_inactive(
self,
test_bot:KleinanzeigenBot,
tmp_path:Path,
state_value:str | None,
) -> None:
"""Test that unexpected state values are treated as inactive."""
test_bot.workspace = xdg_paths.Workspace.for_config(tmp_path / "config.yaml", "kleinanzeigen-bot")
test_bot.ads_selector = "all"
test_bot.browser = MagicMock()
published_ads = [{"id": 123, "state": state_value}] if state_value is not None else [{"id": 123}]
extractor_mock = MagicMock()
extractor_mock.extract_own_ads_urls = AsyncMock(return_value = ["https://www.kleinanzeigen.de/s-anzeige/test/123"])
extractor_mock.extract_ad_id_from_ad_url = MagicMock(return_value = 123)
extractor_mock.navigate_to_ad_page = AsyncMock(return_value = True)
extractor_mock.download_ad = AsyncMock()
with (
patch.object(test_bot, "_fetch_published_ads", AsyncMock(return_value = published_ads)),
patch("kleinanzeigen_bot.extract.AdExtractor", return_value = extractor_mock),
):
await test_bot.download_ads()
# All non-"active" states should result in active=False
extractor_mock.download_ad.assert_awaited_once_with(123, active = False)
class TestKleinanzeigenBotAuthentication:
"""Tests for login and authentication functionality."""
@@ -693,7 +229,7 @@ class TestKleinanzeigenBotAuthentication:
assert await test_bot.is_logged_in() is True
group_text.assert_awaited()
assert any(call.kwargs.get("timeout") == test_bot._timeout("login_detection") for call in group_text.await_args_list)
assert any(call.kwargs.get("timeout") == test_bot.timeout("login_detection") for call in group_text.await_args_list)
@pytest.mark.asyncio
async def test_is_logged_in_runs_full_selector_group_before_cta_precheck(self, test_bot:KleinanzeigenBot) -> None:
@@ -1237,7 +773,7 @@ class TestKleinanzeigenBotAuthentication:
async def test_click_gdpr_banner_uses_quick_dom_timeout_and_clicks_found_element(self, test_bot:KleinanzeigenBot) -> None:
mock_element = AsyncMock()
with (
patch.object(test_bot, "_timeout", return_value = 1.25) as mock_timeout,
patch.object(test_bot, "timeout", return_value = 1.25) as mock_timeout,
patch.object(test_bot, "web_probe", new_callable = AsyncMock, return_value = mock_element) as mock_probe,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock) as mock_sleep,
):
@@ -1253,7 +789,7 @@ class TestKleinanzeigenBotAuthentication:
@pytest.mark.asyncio
async def test_click_gdpr_banner_does_nothing_when_banner_absent(self, test_bot:KleinanzeigenBot) -> None:
with (
patch.object(test_bot, "_timeout", return_value = 1.25),
patch.object(test_bot, "timeout", return_value = 1.25),
patch.object(test_bot, "web_probe", new_callable = AsyncMock, return_value = None) as mock_probe,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock) as mock_sleep,
):
@@ -1266,7 +802,7 @@ class TestKleinanzeigenBotAuthentication:
async def test_dismiss_consent_banner_clicks_when_present(self, test_bot:KleinanzeigenBot) -> None:
mock_element = AsyncMock()
with (
patch.object(test_bot, "_timeout", return_value = 2.0) as mock_timeout,
patch.object(test_bot, "timeout", return_value = 2.0) as mock_timeout,
patch.object(test_bot, "web_probe", new_callable = AsyncMock, return_value = mock_element) as mock_probe,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock) as mock_sleep,
):
@@ -1282,7 +818,7 @@ class TestKleinanzeigenBotAuthentication:
@pytest.mark.asyncio
async def test_dismiss_consent_banner_does_nothing_when_absent(self, test_bot:KleinanzeigenBot) -> None:
with (
patch.object(test_bot, "_timeout", return_value = 2.0),
patch.object(test_bot, "timeout", return_value = 2.0),
patch.object(test_bot, "web_probe", new_callable = AsyncMock, return_value = None) as mock_probe,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock) as mock_sleep,
):
@@ -1295,7 +831,7 @@ class TestKleinanzeigenBotAuthentication:
async def test_check_sms_verification_prompts_user_when_detected(self, test_bot:KleinanzeigenBot) -> None:
mock_element = MagicMock()
with (
patch.object(test_bot, "_timeout", return_value = 3.0) as mock_timeout,
patch.object(test_bot, "timeout", return_value = 3.0) as mock_timeout,
patch.object(test_bot, "web_probe", new_callable = AsyncMock, return_value = mock_element) as mock_probe,
patch("kleinanzeigen_bot.ainput", new_callable = AsyncMock) as mock_ainput,
):
@@ -1310,7 +846,7 @@ class TestKleinanzeigenBotAuthentication:
@pytest.mark.asyncio
async def test_check_sms_verification_returns_silently_when_absent(self, test_bot:KleinanzeigenBot) -> None:
with (
patch.object(test_bot, "_timeout", return_value = 3.0),
patch.object(test_bot, "timeout", return_value = 3.0),
patch.object(test_bot, "web_probe", new_callable = AsyncMock, return_value = None) as mock_probe,
patch("kleinanzeigen_bot.ainput", new_callable = AsyncMock) as mock_ainput,
):
@@ -1323,7 +859,7 @@ class TestKleinanzeigenBotAuthentication:
async def test_check_email_verification_prompts_user_when_detected(self, test_bot:KleinanzeigenBot) -> None:
mock_element = MagicMock()
with (
patch.object(test_bot, "_timeout", return_value = 4.0) as mock_timeout,
patch.object(test_bot, "timeout", return_value = 4.0) as mock_timeout,
patch.object(test_bot, "web_probe", new_callable = AsyncMock, return_value = mock_element) as mock_probe,
patch("kleinanzeigen_bot.ainput", new_callable = AsyncMock) as mock_ainput,
):
@@ -1338,7 +874,7 @@ class TestKleinanzeigenBotAuthentication:
@pytest.mark.asyncio
async def test_check_email_verification_returns_silently_when_absent(self, test_bot:KleinanzeigenBot) -> None:
with (
patch.object(test_bot, "_timeout", return_value = 4.0),
patch.object(test_bot, "timeout", return_value = 4.0),
patch.object(test_bot, "web_probe", new_callable = AsyncMock, return_value = None) as mock_probe,
patch("kleinanzeigen_bot.ainput", new_callable = AsyncMock) as mock_ainput,
):
@@ -1509,7 +1045,7 @@ class TestKleinanzeigenBotBasics:
patch.object(test_bot, "web_request", new_callable = AsyncMock, return_value = {"content": json.dumps(payload)}) as web_request_mock,
patch.object(test_bot, "publish_ad", new_callable = AsyncMock) as publish_ad_mock,
patch.object(test_bot, "web_await", new_callable = AsyncMock, return_value = True) as web_await_mock,
patch.object(test_bot, "delete_ad", new_callable = AsyncMock) as delete_ad_mock,
patch("kleinanzeigen_bot.delete_flow.delete_ad", new_callable = AsyncMock) as delete_ad_mock,
):
await test_bot.publish_ads(ad_cfgs)
@@ -1518,7 +1054,12 @@ class TestKleinanzeigenBotBasics:
web_request_mock.assert_awaited_once_with(expected_url)
publish_ad_mock.assert_awaited_once_with("ad.yaml", ad_cfgs[0][1], {}, [], AdUpdateStrategy.REPLACE)
web_await_mock.assert_awaited_once()
delete_ad_mock.assert_awaited_once_with(ad_cfgs[0][1], [], delete_old_ads_by_title = False)
delete_ad_mock.assert_awaited_once_with(
web = test_bot, root_url = test_bot.root_url,
ad_cfg = ad_cfgs[0][1],
published_ads_list = [],
delete_old_ads_by_title = False,
)
@pytest.mark.asyncio
async def test_publish_ads_uses_millisecond_retry_delay_on_retryable_failure(
@@ -2309,9 +1850,10 @@ class TestKleinanzeigenBotAdOperations:
@pytest.mark.asyncio
async def test_run_download_command_default_selector(self, test_bot:KleinanzeigenBot, mock_config_setup:None) -> None: # pylint: disable=unused-argument
"""Test running download command with default selector."""
with patch.object(test_bot, "download_ads", new_callable = AsyncMock):
with patch("kleinanzeigen_bot.download_flow.download_ads", new_callable = AsyncMock) as mock_download:
await test_bot.run(["script.py", "download"])
assert test_bot.ads_selector == "new"
mock_download.assert_awaited_once()
@pytest.mark.asyncio
async def test_run_update_default_selector(self, test_bot:KleinanzeigenBot, mock_config_setup:None) -> None: # pylint: disable=unused-argument
@@ -2335,9 +1877,10 @@ class TestKleinanzeigenBotAdManagement:
async def test_download_ads_with_specific_ids(self, test_bot:KleinanzeigenBot, mock_config_setup:None) -> None: # pylint: disable=unused-argument
"""Test downloading ads with specific IDs."""
test_bot.ads_selector = "123,456"
with patch.object(test_bot, "download_ads", new_callable = AsyncMock):
with patch("kleinanzeigen_bot.download_flow.download_ads", new_callable = AsyncMock) as mock_download:
await test_bot.run(["script.py", "download", "--ads=123,456"])
assert test_bot.ads_selector == "123,456"
mock_download.assert_awaited_once()
@pytest.mark.asyncio
async def test_run_publish_invalid_selector(self, test_bot:KleinanzeigenBot, mock_config_setup:None) -> None: # pylint: disable=unused-argument
@@ -2368,209 +1911,6 @@ class TestKleinanzeigenBotAdManagement:
assert exc_info.value.code == 2
class TestKleinanzeigenBotAdDeletion:
"""Tests for ad deletion functionality."""
@pytest.mark.asyncio
async def test_delete_ad_by_title_match_succeeds(
self,
test_bot:KleinanzeigenBot,
minimal_ad_config:dict[str, Any],
) -> None:
"""When title matches a published ad and server returns 200, should return True and clear id."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"title": "Test Title", "id": None})
published_ads = [{"title": "Test Title", "id": 67890}, {"title": "Other Title", "id": 11111}]
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock,
return_value = {"statusCode": 200, "statusMessage": "OK", "content": "{}"}),
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await test_bot.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = True)
assert result is True
assert ad_cfg.id is None
@pytest.mark.asyncio
async def test_delete_ad_by_id_succeeds(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When ad has an ID and server returns 200, should return True and clear id."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"id": 12345})
published_ads = [{"title": "Different Title", "id": 12345}, {"title": "Other Title", "id": 11111}]
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock,
return_value = {"statusCode": 200, "statusMessage": "OK", "content": "{}"}),
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await test_bot.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = False)
assert result is True
assert ad_cfg.id is None
@pytest.mark.asyncio
async def test_delete_ad_returns_false_when_no_match(
self,
test_bot:KleinanzeigenBot,
minimal_ad_config:dict[str, Any],
caplog:pytest.LogCaptureFixture,
) -> None:
"""When no published ads match, should return False without opening any pages."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"title": "No Match Title", "id": 99999})
published_ads = [{"title": "Different Title", "id": 12345}]
with (
caplog.at_level(logging.WARNING, logger = "kleinanzeigen_bot"),
patch.object(test_bot, "web_open", new_callable = AsyncMock) as mock_web_open,
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_web_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock) as mock_web_sleep,
patch.object(test_bot, "web_request", new_callable = AsyncMock) as mock_request,
):
result = await test_bot.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = True)
assert result is False
assert ad_cfg.id == 99999 # Preserved — no deletion attempted
mock_web_open.assert_not_called()
mock_web_find.assert_not_called()
mock_web_sleep.assert_not_called()
mock_request.assert_not_called()
@pytest.mark.asyncio
async def test_delete_ad_returns_false_on_404_clears_id(
self,
test_bot:KleinanzeigenBot,
minimal_ad_config:dict[str, Any],
) -> None:
"""When delete is attempted but server returns 404, should return False but still clear the id."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"id": 12345})
published_ads:list[dict[str, Any]] = []
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock, return_value = {"statusCode": 404, "statusMessage": "Not Found", "content": "{}"}),
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await test_bot.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = False)
assert result is False
assert ad_cfg.id is None # Cleared because delete was attempted
@pytest.mark.asyncio
async def test_delete_ad_skips_invalid_published_ad_id(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When a title-matched published ad has an invalid id (None), it should be skipped."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"title": "Test Title", "id": None})
published_ads:list[dict[str, Any]] = [
{"title": "Test Title", "id": None}, # Invalid — should be skipped
{"title": "Test Title", "id": "not-a-number"}, # Invalid — should be skipped
]
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock) as mock_web_open,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock),
):
result = await test_bot.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = True)
assert result is False
mock_web_open.assert_not_called() # No valid IDs → no page open
@pytest.mark.asyncio
async def test_delete_ad_with_zero_id(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When ad_cfg.id is 0 (falsy but valid), should still enter the ID-based deletion path."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"id": 0})
published_ads:list[dict[str, Any]] = []
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock,
return_value = {"statusCode": 200, "statusMessage": "OK", "content": "{}"}) as mock_request,
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await test_bot.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = False)
assert result is True
assert ad_cfg.id is None
mock_request.assert_called_once()
assert "ids=0" in mock_request.call_args[1]["url"]
@pytest.mark.asyncio
async def test_delete_ad_multiple_title_matches(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When multiple published ads match by title with mixed 200/404, should return True."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"title": "Test Title", "id": None})
published_ads = [
{"title": "Test Title", "id": 100},
{"title": "Test Title", "id": 200},
{"title": "Test Title", "id": 300},
]
response_sequence = [
{"statusCode": 200, "statusMessage": "OK", "content": "{}"},
{"statusCode": 404, "statusMessage": "Not Found", "content": "{}"},
{"statusCode": 200, "statusMessage": "OK", "content": "{}"},
]
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock, side_effect = response_sequence) as mock_request,
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await test_bot.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = True)
assert result is True
assert ad_cfg.id is None
assert mock_request.call_count == 3
@pytest.mark.asyncio
async def test_delete_ad_title_and_id_match_deduplicated(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When the same ad matches by both title and ID, it should be deleted only once."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"title": "Test Title", "id": 100})
published_ads = [{"title": "Test Title", "id": 100}]
ok_response = {"statusCode": 200, "statusMessage": "OK", "content": "{}"}
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch.object(test_bot, "web_request", new_callable = AsyncMock, return_value = ok_response) as mock_request,
):
mock_find.return_value.attrs = {"content": "some-token"}
result = await test_bot.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = True)
assert result is True
assert ad_cfg.id is None
mock_request.assert_called_once() # Deduplicated — only one request
@pytest.mark.asyncio
async def test_delete_ad_exception_preserves_id(self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any]) -> None:
"""When web_request raises an exception mid-loop, ad_cfg.id should be preserved and web_sleep not called."""
ad_cfg = Ad.model_validate(minimal_ad_config | {"id": 12345})
published_ads:list[dict[str, Any]] = []
with (
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "web_find", new_callable = AsyncMock) as mock_find,
patch.object(test_bot, "web_sleep", new_callable = AsyncMock) as mock_web_sleep,
patch.object(test_bot, "web_request", new_callable = AsyncMock, side_effect = TimeoutError("request timed out")),
):
mock_find.return_value.attrs = {"content": "some-token"}
with pytest.raises(TimeoutError, match = "request timed out"):
await test_bot.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = False)
assert ad_cfg.id == 12345 # Preserved — exception prevented clearing
mock_web_sleep.assert_not_called()
class TestKleinanzeigenBotShippingOptions:
"""Tests for shipping options functionality."""
@@ -2738,7 +2078,7 @@ class TestKleinanzeigenBotShippingOptions:
patch("kleinanzeigen_bot.Path", PureWindowsPath),
patch("kleinanzeigen_bot.price_reduction.apply_auto_price_reduction", side_effect = mock_apply_auto_price_reduction),
patch.object(test_bot, "web_open", new_callable = AsyncMock),
patch.object(test_bot, "delete_ad", new_callable = AsyncMock),
patch("kleinanzeigen_bot.delete_flow.delete_ad", new_callable = AsyncMock),
):
# Call publish_ad and expect sentinel exception
try:
@@ -2794,6 +2134,7 @@ class TestKleinanzeigenBotShippingOptions:
with (
patch("kleinanzeigen_bot.price_reduction.apply_auto_price_reduction") as mock_apply,
patch("kleinanzeigen_bot.delete_flow.delete_ad", new_callable = AsyncMock),
patch.object(test_bot, "web_probe", new_callable = AsyncMock, side_effect = mock_web_probe),
patch.object(test_bot, "web_find", new_callable = AsyncMock),
patch.object(test_bot, "web_input", new_callable = AsyncMock),
@@ -4993,59 +4334,3 @@ class TestTrackingFallback:
result = await test_bot._try_recover_ad_id_from_redirect()
assert result == 11223344
class TestDeleteAdsAfterDeletePolicy:
"""Tests for delete_ads orchestration with after_delete policy integration."""
@staticmethod
def _make_ad(minimal_ad_config:dict[str, Any], tmp_path:Path) -> tuple[str, Ad, dict[str, Any]]:
ad_cfg = Ad.model_validate(minimal_ad_config | {
"id": 12345, "active": True,
"created_on": "2024-06-01T12:00:00", "updated_on": "2024-06-10T08:30:00",
"content_hash": "abc123", "repost_count": 3, "price_reduction_count": 1,
})
return str(tmp_path / "ad.yaml"), ad_cfg, ad_cfg.model_dump()
@pytest.mark.asyncio
async def test_cleanup_on_404_detection(
self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any], tmp_path:Path,
) -> None:
"""Cleanup runs when delete_ad returns False but cleared the id (404 path)."""
test_bot.config.deleting.after_delete = "RESET"
ad_file, ad_cfg, ad_cfg_orig = self._make_ad(minimal_ad_config, tmp_path)
async def fake_delete(ad:Ad, _published:list[dict[str, Any]], **__:Any) -> bool:
ad.id = None # Phase B ran and cleared the id
return False # but all responses were 404
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch.object(test_bot, "delete_ad", new_callable = AsyncMock, side_effect = fake_delete),
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch("kleinanzeigen_bot.utils.dicts.save_dict") as mock_save,
):
await test_bot.delete_ads([(ad_file, ad_cfg, ad_cfg_orig)])
assert ad_cfg.repost_count == 0
assert "id" not in ad_cfg_orig
mock_save.assert_called_once()
@pytest.mark.asyncio
async def test_no_cleanup_when_delete_not_attempted(
self, test_bot:KleinanzeigenBot, minimal_ad_config:dict[str, Any], tmp_path:Path,
) -> None:
"""No cleanup when delete_ad returns False with id preserved (no match)."""
test_bot.config.deleting.after_delete = "RESET"
ad_file, ad_cfg, ad_cfg_orig = self._make_ad(minimal_ad_config, tmp_path)
with (
patch.object(test_bot, "_fetch_published_ads", new_callable = AsyncMock, return_value = []),
patch.object(test_bot, "delete_ad", new_callable = AsyncMock, return_value = False),
patch.object(test_bot, "web_sleep", new_callable = AsyncMock),
patch("kleinanzeigen_bot.utils.dicts.save_dict") as mock_save,
):
await test_bot.delete_ads([(ad_file, ad_cfg, ad_cfg_orig)])
mock_save.assert_not_called()
assert ad_cfg.id == 12345

74
tests/unit/test_json_pagination.py → tests/unit/test_published_ads.py
View File

@@ -1,4 +1,4 @@
# SPDX-FileCopyrightText: © Sebastian Thomschke and contributors
# SPDX-FileCopyrightText: © Jens Bergmann and contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
# SPDX-ArtifactOfProjectHomePage: https://github.com/Second-Hand-Friends/kleinanzeigen-bot/
"""Tests for JSON API pagination helper methods."""
@@ -8,14 +8,14 @@ from unittest.mock import AsyncMock, patch
import pytest
from kleinanzeigen_bot import KleinanzeigenBot
from kleinanzeigen_bot import KleinanzeigenBot, published_ads
from kleinanzeigen_bot.published_ads import PublishedAdsFetchIncompleteError
from kleinanzeigen_bot.utils import misc
from kleinanzeigen_bot.utils.exceptions import PublishedAdsFetchIncompleteError
@pytest.mark.unit
class TestJSONPagination:
"""Tests for _coerce_page_number and _fetch_published_ads methods."""
"""Tests for coerce_page_number and fetch_published_ads methods."""
@pytest.fixture
def bot(self) -> KleinanzeigenBot:
@@ -97,7 +97,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": '{"ads": [{"id": 1, "state": "active", "title": "Ad 1"}, {"id": 2, "state": "active", "title": "Ad 2"}]}'}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if len(result) != 2:
pytest.fail(f"Expected 2 results, got {len(result)}")
@@ -115,7 +115,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": json.dumps(response_data)}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if len(result) != 1:
pytest.fail(f"Expected 1 ad, got {len(result)}")
@@ -137,7 +137,7 @@ class TestJSONPagination:
{"content": json.dumps(page3_data)},
]
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if len(result) != 6:
pytest.fail(f"Expected 6 ads but got {len(result)}")
@@ -157,12 +157,12 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": json.dumps(response_data)}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if not isinstance(result, list):
pytest.fail(f"expected result to be list, got {type(result).__name__}")
if len(result) != 0:
pytest.fail(f"expected empty list from _fetch_published_ads, got {len(result)} items")
pytest.fail(f"expected empty list from fetch_published_ads, got {len(result)} items")
@pytest.mark.asyncio
async def test_fetch_published_ads_invalid_json(self, bot:KleinanzeigenBot) -> None:
@@ -170,7 +170,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": "invalid json"}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if result != []:
pytest.fail(f"Expected empty list on invalid JSON, got {result}")
@@ -182,7 +182,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": json.dumps(response_data)}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if len(result) != 2:
pytest.fail(f"expected 2 ads, got {len(result)}")
@@ -196,7 +196,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": json.dumps(response_data)}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
# Should return ads from first page and stop due to invalid paging
if len(result) != 1:
@@ -212,7 +212,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": json.dumps(response_data)}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
# Should return empty list when ads is not a list
if not isinstance(result, list):
@@ -228,7 +228,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": json.dumps(response_data)}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if result != [{"id": 1, "state": "active"}]:
pytest.fail(f"expected malformed entries to be filtered out, got: {result}")
@@ -252,7 +252,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": json.dumps(response_data)}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if result != [{"id": 2, "state": "paused"}]:
pytest.fail(f"expected only entries with id and state to remain, got: {result}")
@@ -267,7 +267,7 @@ class TestJSONPagination:
patch.object(bot, "web_request", mock_request),
pytest.raises(PublishedAdsFetchIncompleteError, match = "Filtered 2 malformed ad entries on page 1"),
):
await bot._fetch_published_ads(strict = True)
await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url, strict = True)
@pytest.mark.asyncio
async def test_fetch_published_ads_timeout(self, bot:KleinanzeigenBot) -> None:
@@ -275,7 +275,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.side_effect = TimeoutError("timeout")
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if result != []:
pytest.fail(f"Expected empty list on timeout, got {result}")
@@ -287,7 +287,7 @@ class TestJSONPagination:
patch.object(bot, "web_request", new_callable = AsyncMock, side_effect = TimeoutError("timeout")),
pytest.raises(PublishedAdsFetchIncompleteError, match = "Pagination request failed on page 1"),
):
await bot._fetch_published_ads(strict = True)
await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url, strict = True)
@pytest.mark.asyncio
async def test_fetch_published_ads_handles_non_string_content_type(self, bot:KleinanzeigenBot) -> None:
@@ -295,7 +295,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": None}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if result != []:
pytest.fail(f"expected empty result on non-string content, got: {result}")
@@ -312,7 +312,7 @@ class TestJSONPagination:
{"content": json.dumps(page2)},
]
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if [ad["id"] for ad in result] != [1, 2, 3]:
pytest.fail(f"Expected ids [1, 2, 3] but got {[ad['id'] for ad in result]}")
@@ -333,7 +333,7 @@ class TestJSONPagination:
with patch.object(bot, "web_request", new_callable = AsyncMock) as mock_request:
mock_request.return_value = {"content": json.dumps(response_data)}
result = await bot._fetch_published_ads()
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url)
if len(result) != 2:
pytest.fail(f"Expected 2 ads, got {len(result)}")
@@ -344,8 +344,8 @@ class TestJSONPagination:
)
@pytest.mark.asyncio
async def test_fetch_published_ads_single_page_no_last_no_next_strict_raises(self, bot:KleinanzeigenBot) -> None:
"""Strict mode should fail when paging omits both 'last' and 'next'."""
async def test_fetch_published_ads_single_page_no_last_no_next_strict_returns_ads(self, bot:KleinanzeigenBot) -> None:
"""Strict mode should NOT raise when a single page has no 'next' and no total_pages — it is a complete response."""
response_data = {
"ads": [{"id": 10, "state": "active"}],
"paging": {"pageNum": 1},
@@ -353,6 +353,30 @@ class TestJSONPagination:
with (
patch.object(bot, "web_request", new_callable = AsyncMock, return_value = {"content": json.dumps(response_data)}),
pytest.raises(PublishedAdsFetchIncompleteError, match = r"No 'next' in paging on page 1"),
):
await bot._fetch_published_ads(strict = True)
result = await published_ads.fetch_published_ads(web = bot, root_url = bot.root_url, strict = True)
if len(result) != 1:
pytest.fail(f"Expected 1 ad in strict mode single-page response, got {len(result)}")
if result[0]["id"] != 10:
pytest.fail(f"Expected ad id 10, got {result[0]['id']}")
def test_ad_matches_id(self) -> None:
"""Shared helper for safe API-ID comparison, used by publish/update/extend/delete."""
# Matching int to int
assert published_ads.ad_matches_id({"id": 42}, 42) is True
# Matching str to int (API returns string IDs)
assert published_ads.ad_matches_id({"id": "42"}, 42) is True
# Mismatch
assert published_ads.ad_matches_id({"id": 99}, 42) is False
assert published_ads.ad_matches_id({"id": "99"}, 42) is False
# target_id is None → always False
assert published_ads.ad_matches_id({"id": 42}, None) is False
# ad has no "id" key
assert published_ads.ad_matches_id({}, 42) is False
# ad has "id": None
assert published_ads.ad_matches_id({"id": None}, 42) is False
# Unparseable id (not coercible to int)
assert published_ads.ad_matches_id({"id": "nope"}, 42) is False
# Boolean (True → 1) — accepting bool-to-int coercion is current behavior
assert published_ads.ad_matches_id({"id": True}, 1) is True

3
tests/unit/test_web_scraping_mixin.py
View File

@@ -812,7 +812,8 @@ class TestSelectorTimeoutMessages:
assert result is element
once_call = once_mock.await_args_list[0]
assert once_call.args[:2] == (By.ID, "probe-id")
assert once_call.args[2] == web_scraper._timeout("quick_dom")
assert isinstance(once_call.args[2], (int, float))
assert once_call.args[2] > 0
retry_mock.assert_not_awaited()
@pytest.mark.asyncio

28
tests/unit/test_web_scraping_pagination.py
View File

@@ -1,7 +1,7 @@
# SPDX-FileCopyrightText: © Sebastian Thomschke and contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
# SPDX-ArtifactOfProjectHomePage: https://github.com/Second-Hand-Friends/kleinanzeigen-bot/
"""Tests for the _navigate_paginated_ad_overview helper method."""
"""Tests for the navigate_paginated_ad_overview helper method."""
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
@@ -12,7 +12,7 @@ from kleinanzeigen_bot.utils.web_scraping_mixin import By, Element, WebScrapingM
class TestNavigatePaginatedAdOverview:
"""Tests for _navigate_paginated_ad_overview method."""
"""Tests for navigate_paginated_ad_overview method."""
@staticmethod
async def _single_page_find_side_effect(selector_type:By, selector_value:str, **kwargs:Any) -> Element: # noqa: ARG004
@@ -34,9 +34,9 @@ class TestNavigatePaginatedAdOverview:
patch.object(mixin, "web_find", new_callable = AsyncMock, side_effect = self._single_page_find_side_effect),
patch.object(mixin, "web_find_all", new_callable = AsyncMock, side_effect = TimeoutError("No pagination")),
patch.object(mixin, "web_scroll_page_down", new_callable = AsyncMock),
patch.object(mixin, "_timeout", return_value = 10),
patch.object(mixin, "timeout", return_value = 10),
):
result = await mixin._navigate_paginated_ad_overview(callback)
result = await mixin.navigate_paginated_ad_overview(callback)
assert result is True
callback.assert_awaited_once_with(1)
@@ -55,9 +55,9 @@ class TestNavigatePaginatedAdOverview:
patch.object(mixin, "web_find", new_callable = AsyncMock, side_effect = self._single_page_find_side_effect),
patch.object(mixin, "web_find_all", new_callable = AsyncMock, side_effect = TimeoutError("No pagination")),
patch.object(mixin, "web_scroll_page_down", new_callable = AsyncMock),
patch.object(mixin, "_timeout", return_value = 10),
patch.object(mixin, "timeout", return_value = 10),
):
result = await mixin._navigate_paginated_ad_overview(callback)
result = await mixin.navigate_paginated_ad_overview(callback)
assert result is False
callback.assert_awaited_once_with(1)
@@ -86,9 +86,9 @@ class TestNavigatePaginatedAdOverview:
patch.object(mixin, "web_find", new_callable = AsyncMock, side_effect = self._single_page_find_side_effect),
patch.object(mixin, "web_find_all", new_callable = AsyncMock, side_effect = mock_find_all_side_effect),
patch.object(mixin, "web_scroll_page_down", new_callable = AsyncMock),
patch.object(mixin, "_timeout", return_value = 10),
patch.object(mixin, "timeout", return_value = 10),
):
result = await mixin._navigate_paginated_ad_overview(callback)
result = await mixin.navigate_paginated_ad_overview(callback)
assert result is True
assert callback.await_count == 2
@@ -102,7 +102,7 @@ class TestNavigatePaginatedAdOverview:
callback = AsyncMock()
with patch.object(mixin, "web_open", new_callable = AsyncMock, side_effect = TimeoutError("Page load timeout")):
result = await mixin._navigate_paginated_ad_overview(callback)
result = await mixin.navigate_paginated_ad_overview(callback)
assert result is False
callback.assert_not_awaited() # Callback should not be called
@@ -119,7 +119,7 @@ class TestNavigatePaginatedAdOverview:
patch.object(mixin, "web_sleep", new_callable = AsyncMock),
patch.object(mixin, "web_find", new_callable = AsyncMock, side_effect = TimeoutError("Container not found")),
):
result = await mixin._navigate_paginated_ad_overview(callback)
result = await mixin.navigate_paginated_ad_overview(callback)
assert result is False
callback.assert_not_awaited()
@@ -137,9 +137,9 @@ class TestNavigatePaginatedAdOverview:
patch.object(mixin, "web_find", new_callable = AsyncMock, side_effect = self._single_page_find_side_effect),
patch.object(mixin, "web_find_all", new_callable = AsyncMock, side_effect = TimeoutError("No pagination")),
patch.object(mixin, "web_scroll_page_down", new_callable = AsyncMock, side_effect = TimeoutError("Scroll timeout")),
patch.object(mixin, "_timeout", return_value = 10),
patch.object(mixin, "timeout", return_value = 10),
):
result = await mixin._navigate_paginated_ad_overview(callback)
result = await mixin.navigate_paginated_ad_overview(callback)
# Should continue and call callback despite scroll timeout
assert result is True
@@ -158,9 +158,9 @@ class TestNavigatePaginatedAdOverview:
patch.object(mixin, "web_find", new_callable = AsyncMock, side_effect = self._single_page_find_side_effect),
patch.object(mixin, "web_find_all", new_callable = AsyncMock, side_effect = TimeoutError("No pagination")),
patch.object(mixin, "web_scroll_page_down", new_callable = AsyncMock),
patch.object(mixin, "_timeout", return_value = 10),
patch.object(mixin, "timeout", return_value = 10),
):
result = await mixin._navigate_paginated_ad_overview(callback)
result = await mixin.navigate_paginated_ad_overview(callback)
assert result is False
callback.assert_awaited_once_with(1)