mirror of
https://github.com/Second-Hand-Friends/kleinanzeigen-bot.git
synced 2026-07-09 20:51:05 +02:00
a051712dc8ae5ef159e56b30c233a14155c5da77
34 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Alexander Peltzer
|
89001b5080 |
feat: Add human-like interactions (#1193)
## ℹ️ Description Adds configurable human-like browser interaction behavior to make automation less mechanically deterministic while preserving the existing publish/update workflows. Related to #1120. ## 📋 Changes Summary - Added `humanization` configuration for action delays, typing jitter, thinking pauses, idle micro-actions, and randomized viewport sizes. - Applied humanized timing and interaction behavior to relevant browser automation paths. - Added validation for viewport size formats and min/max timing ranges, including German translations for user-facing validation errors. - Hardened fallback behavior for humanized click and typing flows so automation can continue when a lower-level interaction fails. - Added focused unit coverage for delay edge cases, config validation, and typing fallback behavior. - Regenerated committed config/schema artifacts for the new settings. ## 🧪 Validation - `pdm run pytest tests/unit/test_humanization.py tests/unit/test_web_scraping_mixin.py` - `pdm run ruff check src/kleinanzeigen_bot/utils/web_scraping_mixin.py src/kleinanzeigen_bot/model/config_model.py tests/unit/test_humanization.py` - Live DOM probes with the PR worktree code via `data/verify_dom_assumptions.py`: - `login-selectors` - `publish-selectors` - `price-type-controls` ### ⚙️ Type of Change - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [x] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable “humanization” behavior for browser automation: optional mouse movement, per-keystroke typing jitter/delays, probabilistic thinking pauses, idle micro-actions, and optional launch viewport randomization. * Navigation timing now incorporates human-like action delays. * **Bug Fixes** * Improved resilience: humanized click/typing safely fall back to standard actions when needed. * Added stronger validation for viewport-size formatting and delay min/max range consistency. * **Tests** * Added comprehensive unit coverage for humanization routing, event dispatch, timing, fallbacks, idle/thinking behavior, viewport launch arguments, and validation. * **Documentation** * Updated default configuration, configuration schema, and a related translation message. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Jens <1742418+1cu@users.noreply.github.com> |
||
Jens
|
d384f172b8 |
fix: remove individual shipping publishing (#1159)
## ℹ️ Description - Link to the related issue(s): Issue #1157 - Kleinanzeigen no longer offers individual/custom shipping. This updates publishing, validation, docs, schemas, translations, and tests so existing `shipping_costs` configs are handled gracefully instead of failing against removed DOM controls. ## 📋 Changes Summary - Keep `shipping_costs` for legacy configs/downloaded ads, but mark it deprecated in schema/docs and ignore it during publishing. - Remove publishing interaction with removed individual-shipping DOM controls. - Require predefined non-empty `shipping_options` for `sell_directly`. - Update docs, generated schemas/config artifacts, German translations, and unit tests. - Verified live DOM for `sell_directly: true` + `shipping_type: SHIPPING` + predefined `shipping_options` via local ignored script; no ad submission performed. ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [x] 🐞 Bug fix (non-breaking change which fixes an issue) - [ ] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated shipping guides to deprecate custom individual shipping costs and make publishing depend on per-ad predefined `shipping_options` (DHL/Hermes; selectable options must come from a single size group). * Added migration guidance replacing `shipping_costs` with the correct `shipping_options`. * Clarified “Sell directly” requires `shipping_type: SHIPPING`, non-empty `shipping_options`, and `FIXED`/`NEGOTIABLE` pricing. * **Bug Fixes** * Publishing is blocked when `shipping_costs` is provided without `shipping_options`. * Removed unsupported individual-shipping handling and tightened “Sell directly” eligibility rules. * **Tests** * Expanded unit coverage for deprecation behavior, content handling, and direct-buy validation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
f1e37c6f58 |
fix: disable diagnostics timing collection by default (#1115)
## ℹ️ Description Change the `diagnostics.timing_collection` default from `true` to `false` so that timeout timing data is only collected when a user explicitly opts in. ## 📋 Changes Summary - `model/config_model.py`: changed `TimingCollectionConfig.timing_collection` default from `True` to `False` - `schemas/config.schema.json`: regenerated with `"default": false` - `docs/config.default.yaml`: regenerated with `timing_collection: false` Existing configs that explicitly set `timing_collection: true` are unaffected. ### ⚙️ Type of Change - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [ ] ✨ New feature (adds new functionality without breaking existing usage) - [x] 💥 Breaking change (changes default behavior — users who relied on the implicit `true` default must now explicitly enable it) ## ✅ Checklist - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. |
||
|
Jens
|
3958d4c907 |
feat: add preserve-local-settings toggle for download (#1091)
## ℹ️ Description When re-downloading an ad that is already saved locally (via `--ads=all` or numeric IDs), the ad YAML is atomically replaced. The download only scrapes fields from the live page, so all **local-only settings** are reset to defaults or zero: | Field | What happens | |---|---| | `auto_price_reduction` | Reset to `AdDefaults` value (or `null`) | | `republication_interval` | Reset to `AdDefaults` value (or `null`) | | `repost_count` | Reset to `0` — bot forgets how many times it published | | `price_reduction_count` | Reset to `0` — bot forgets auto-reduction progress | The `new` selector avoids this by skipping already-saved ads, but `all` and numeric IDs do not. - Link to the related issue(s): N/A ## 📋 Changes Summary - Add `download.preserve_local_settings` config option (default: `true`) - Add `--preserve-local-settings` CLI flag with help text (English + German) - In `download_ad()`, when re-downloading an existing ad with the toggle enabled, preserve four local-only fields from the existing YAML: `auto_price_reduction`, `republication_interval`, `repost_count`, `price_reduction_count` - Uses validate-then-commit pattern: candidate values are staged on a model copy, validated, then committed only on success - Added test verifying local fields survive re-download - Regenerated schemas and config defaults - Updated German translations for new log messages ### ⚙️ Type of Change - [x] 🐞 Bug fix - [x] ✨ New feature - [ ] 💥 Breaking change ## ✅ Checklist - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added `--preserve-local-settings` CLI flag to retain local ad settings during re-downloads (including auto price reduction, republication interval, and repost count). * Settings now preserved by default; re-downloading existing ads maintains local configurations instead of resetting to defaults. * **Documentation** * Updated configuration guide with details on the new `preserve_local_settings` option. * **Tests** * Added unit tests for CLI flag parsing and preservation logic during ad re-downloads. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
c872621f28 |
feat: add opt-in local path renaming after ad republish (#1056)
## ℹ️ Description Closes: Discussion [#839](https://github.com/Second-Hand-Friends/kleinanzeigen-bot/discussions/839) When an ad is re-published on Kleinanzeigen, it receives a new ad ID — but local files and folders still carry the old ID in their names (e.g. `ad_5195800653_Title/ad_5195800653.yaml`). Over time, file names become stale and locating ads in the filesystem becomes confusing. This PR adds an opt-in, conservative path renaming feature that runs after a successful publish/republish. By default nothing changes (`OFF`). Users can enable `TEMPLATE_MATCH` mode, which renames only paths whose names match the structure of their configured download naming templates — preserving user-edited titles and never touching manually-named or unrelated files. ## Design decisions - **Opt-in and backwards-compatible**: default `OFF`; no existing setups are affected. - **Config lives under `publishing`** (not `download`): the trigger is a successful publish with a changed ad ID. - **Template-derived matching, not full re-render**: `{title}` is matched as wildcard (preserving user edits and truncation), while only the numeric ID in the `{id}` slot is replaced. This avoids brittleness from title changes or truncation drift, and handles the case where a prior republish already left a stale ID in the path. - **Uses existing download templates**: `download.folder_name_template` and `download.ad_file_name_template` serve as provenance hints. Changing those templates automatically changes which paths qualify for renaming — no separate renaming config needed. - **Conservative scope**: only the loaded ad file, its parent folder, and its referenced images are touched. ## How it works (per ad, after successful publish) 1. **Referenced image files are renamed first**, and the YAML `images` list is updated with new paths — *before* `content_hash` is recomputed and the YAML is saved. This ensures the saved file references actually-existing files and the hash matches. 2. **The ad YAML and containing folder are renamed after saving**, since the saved file itself may move. 3. **If saving fails, image renames are rolled back** to avoid leaving the filesystem in an inconsistent state. ## Safety guardrails | Guardrail | Behavior | |---|---| | Name mismatch | Skip — the helper only matches names whose structure conforms to the configured download templates | | Target exists (file or symlink) | Skip — never overwrite; log a warning | | Absolute image paths | Skip — cannot be safely remapped after a folder rename | | Paths outside the ad folder (`../`) | Skip — avoid touching external/shared files | | Idempotent (`old_id == new_id`) | Skip — no rename needed | | Mode `OFF` (default) | Entire feature is a no-op | | Save failure after image renames | Image renames are rolled back before re-raising | ## 📋 Changes Summary ### Configuration - **New**: `publishing.local_path_renaming.mode` (`"OFF"` | `"TEMPLATE_MATCH"`, default `"OFF"`) - Description documents the interaction with `download.folder_name_template` / `download.ad_file_name_template` ### Core logic (`src/kleinanzeigen_bot/__init__.py`) - `RenameStatus` enum and `RenamePathResult` / `ImageRenameResult` / `LocalPathRenameResult` dataclasses — structured result types instead of booleans/tuples - `_replace_template_id_slot()` — slot-aware regex matcher; replaces only the `{id}` value, preserves everything else - `_rename_path_if_target_is_free()` — rename with collision/symlink safety - `_rename_local_ad_file_and_folder_after_id_change()` — renames YAML file and parent folder - `_rename_referenced_local_image_files_after_id_change()` — renames image files referenced in `ad_cfg_orig["images"]`, returns updated image refs without mutating the config dict - `_log_local_path_rename_result()` — structured summary logging with translated labels ### Generated artifacts - `docs/config.default.yaml` — new `publishing.local_path_renaming.mode: OFF` section - `schemas/config.schema.json` — updated ### Translations - German translations added for all new log messages and dynamic labels ### Tests - Unit tests covering: config validation, template matching (multiple templates), disabled-by-default, image path updates, unreferenced images ignored, collision handling, absolute/external paths skipped, broken symlink collision, save-failure rollback, cross-platform path separators ## ⚙️ Type of Change - [x] ✨ New feature (adds new functionality without breaking existing usage) ## ✅ Checklist - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test` → 1119 passed, 4 skipped). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary (generated artifacts). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Template-driven local renaming of ad YAML, containing folder, and referenced relative images when an ad ID changes; only the {id} portion is updated, collisions or unsafe targets are skipped, and a human-readable summary is logged. * **Configuration** * New publishing.local_path_renaming option with OFF (default) and TEMPLATE_MATCH modes to enable template-based renaming. * **Bug Fixes** * Image renames are rolled back if saving fails; publish flow ordering adjusted to ensure consistent local renames. * **Tests** * Added unit tests for template matching, id-slot substitution, image-path updates, collision/symlink safety, and scope rules. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/Second-Hand-Friends/kleinanzeigen-bot/pull/1056?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
ed3b73c3fb |
feat: resolve login credentials from environment variables (#1061)
## ℹ️ Description Adds support for `${KLEINANZEIGEN_BOT_USERNAME}` and `${KLEINANZEIGEN_BOT_PASSWORD}` environment variable substitution in `login.username` and `login.password` config values. Also supports `${VAR:-default}` fallback syntax. This allows users to commit `config.yaml` to VCS without exposing credentials. Closes #1060. ## 📋 Changes Summary - New `_resolve_login_credentials()` static method on `KleinanzeigenBot` — resolves `${VAR}` and `${VAR:-default}` patterns from `os.environ` for `login.username` and `login.password` only - Called in `load_config()` after YAML loading, before Pydantic validation — `LoginConfig` model remains unchanged (`Field(..., min_length=1)`) - Module-level `_LOGIN_ENV_PATTERN` regex for the `${VAR}` / `${VAR:-default}` pattern - Guard against non-dict YAML input (passes through to Pydantic for clean validation errors) - Updated `README.md` and `docs/CONFIGURATION.md` with env var examples - Removed obsolete "Config file not found" translation (no longer raised) - 9 new unit tests in `TestResolveLoginCredentials` covering resolution, fallback, error cases, edge cases, and `load_config` integration ### ⚙️ Type of Change - [x] ✨ New feature (adds new functionality without breaking existing usage) ## ✅ Checklist - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Login credentials support environment variable substitution using ${VARIABLE} with optional fallbacks (${VARIABLE:-default}); plaintext values remain supported. * **Documentation** * Configuration docs updated with YAML and shell examples (exporting variables) and an explicit security note advising against committing plaintext credentials. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/Second-Hand-Friends/kleinanzeigen-bot/pull/1061?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
96734cc5ed |
docs: clarify download naming examples (#1016)
## ℹ️ Description Documentation-only update to clarify how `ad_files` differs from download naming templates and how published IDs are handled. - Link to the related discussion: https://github.com/Second-Hand-Friends/kleinanzeigen-bot/discussions/839 - Describe the motivation and context for this change: reduce confusion around downloaded ad naming and local file behavior. ## 📋 Changes Summary - Clarified `ad_files` as a file-selection glob, separate from download naming. - Added concrete sample ID/title render examples for folder and file templates. - Documented that publish updates the YAML `id` in place and does not auto-rename local files or folders. - Regenerated config docs and schema artifacts to keep generated output in sync. ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [ ] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Expanded ad configuration and setup documentation with improved guidance on file pattern matching and download directory configuration. * Clarified how placeholder templates work in file naming, including required and optional parameters. * Added comprehensive examples demonstrating template rendering and configuration best practices. * Documented republish behavior and workflow guidance for managing downloaded ad files. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
b2970eb4f0 |
feat(delete): add after_delete config for post-delete YAML cleanup (#1000)
## ℹ️ Description - Link to the related issue(s): Issue #994 - Adds a `deleting.after_delete` config toggle that controls what happens to the local ad YAML file after the standalone `delete` command runs. Three modes: `NONE` (default, no change), `RESET` (clear server-side metadata so the ad is eligible for republication as "new"), `DISABLE` (set `active: false` so the ad is skipped until manually re-enabled). ## 📋 Changes Summary - Added `DeletingConfig` model with `after_delete: Literal["NONE", "RESET", "DISABLE"]` field (default `"NONE"`) - Added `deleting: DeletingConfig` field to `Config` class - Added `_apply_after_delete_policy()` helper that mutates `Ad` model and dict in-place based on mode - Integrated policy in `delete_ads` with detection heuristic: cleanup runs when `delete_ad` returns `True` (200) or when `ad_cfg.id` transitions from non-None to None (404) - Policy does NOT apply during publish-time `delete_ad` call sites — only the standalone delete command - Regenerated `docs/config.default.yaml` and `schemas/config.schema.json` ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [x] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable post-delete cleanup options to control how local ad records are handled after deletion (leave, reset metadata, or disable). * **Bug Fixes** * More accurate detection of attempted vs. successful deletions and correct incrementing of deletion counts. * Ensure metadata cleanup is skipped when no targets are found while still handling 404-style removals appropriately. * **Tests** * Added unit tests covering post-delete behaviours and edge cases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
a13fb4f106 |
feat(timing): tune default timeouts and migrate optional probes to web_probe (#985)
## ℹ️ Description - Link to the related issue(s): Issue # - Analyzed 524 timing records across 18 sessions from real-world usage to identify genuine timeout failures vs expected element-absence probes. Three timeout defaults were too tight for observed latency spikes, and four call sites were wasting time on retry/backoff for elements that are intentionally absent in most flows. ## 📋 Changes Summary ### Timeout default changes (`config_model.py` → generated `config.default.yaml` + `config.schema.json`) | Key | Old | New | Rationale | |---|---|---|---| | `sms_verification` | 4.0 | **5.0** | ~11% margin over observed 4.46s failure | | `email_verification` | 4.0 | **5.0** | ~23% margin over observed 4.05s failure | | `login_detection` | 10.0 | **12.0** | ~11% margin over observed 10.6–10.8s failures | | `captcha_detection` | 2.0 | unchanged | No evidence of benefit from timing data | ### Call-site migration (`__init__.py`) Migrated 4 optional-probe methods from `web_find` (retry-backed, raises `TimeoutError`) to `web_probe` (single attempt, returns `None`): - **`_check_sms_verification`**: `web_find` → `web_probe` with explicit `sms_verification` timeout, returns early with debug log when absent - **`_check_email_verification`**: Same pattern with `email_verification` timeout - **`_dismiss_consent_banner`**: Eliminated double-lookup (`web_find` + `web_click`) → single `web_probe` + `element.click()` + `web_sleep()` - **`_click_gdpr_banner`**: Same double-lookup elimination, added debug log on absent path Removed 4 `try/except TimeoutError` wrappers from callers (`handle_after_login_logic` and `login`) since `web_probe` never raises `TimeoutError`. ### Performance impact - Worst-case wait on absent elements reduced from ~19s → 5s (SMS/email verification with retry/backoff vs single probe) - GDPR banner clicks: double DOM lookup eliminated → single lookup ### Test cleanup (`test_init.py`) - **Added 8 behavioral tests** for `_dismiss_consent_banner` (found + absent), `_check_sms_verification` (found + absent), `_check_email_verification` (found + absent), and `_click_gdpr_banner` absent path - **Removed 3 low-value tests**: `test_handle_after_login_logic` (structural mock wiring), `test_is_logged_in_returns_true_with_alternative_element` (duplicate branch), `test_wait_for_post_auth0_submit_transition_sleep_fallback_when_login_not_confirmed` (duplicate outcome) - **Upgraded 1 test**: `test_fill_login_data_and_send` from mock-count assertions to behavioral argument assertions verifying correct selectors and credentials ### ⚙️ Type of Change - [x] 🐞 Bug fix (non-breaking change which fixes an issue) - [ ] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Post-login flows now safely handle absent SMS/email prompts and consent/GDPR banners, avoiding interruptions and clicking banners when present. * **Improvements** * Increased default timeouts for SMS verification, email verification, and login detection to improve login reliability. * **Documentation** * Configuration docs now reference authoritative default timeout values instead of embedding examples. * **Tests** * Strengthened unit tests for login interactions, verification prompts, and banner dismissal behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
c3584d0b2e |
feat(pricing): support update-mode reductions and verify previews (#970)
## ℹ️ Description Automatic price reduction now works reliably during `update` runs (MODIFY mode), not only during `publish`, while staying backward compatible for existing configurations. The verify flow also explains what will happen in both modes before any command touches live ads. - Link to the related issue(s): Issue # N/A - Describe the motivation and context for this change. - Update-mode reductions needed to be explicitly opt-in so existing setups remain unchanged by default. - Pricing behavior had to stay deterministic across retries and mixed publish/update workflows. - Preview output and documentation now mirror real runtime behavior so configuration decisions are easier to trust. ## 📋 Changes Summary - Added `auto_price_reduction.on_update` (`bool`, default `false`) in model/schema so update-mode reductions are explicitly opt-in. - Refactored price reduction flow into decision + apply steps: - `evaluate_auto_price_reduction(...)` computes mode-aware outcomes - `apply_auto_price_reduction(...)` applies mutations and logs from that decision - Implemented restore-first behavior so effective reduced price is recomputed from base `price` + `price_reduction_count` before deciding a new cycle. - Updated `verify` to preview both publish and update outcomes. - Kept update-mode semantics explicit: - `delay_days` applies - `delay_reposts` is ignored in MODIFY mode - Fixed retry idempotency in `publish_ads` by restoring baseline `price` and `price_reduction_count` before each retry. - Addressed follow-up review feedback: - no-visible-change cycles advance counters (for fractional accumulation) - verify output stays non-misleading for no-visible-change cases - tightened a unit assertion from `>= 1` to `== 1` - tightened smoke behavior for `create-config` when config already exists - clarified README/docs and documented `price_reduction_count` as auto-managed - Expanded/updated tests for: - MODIFY `on_update` behavior - restore-first invariant - cross-mode interactions - retry baseline reset - verify mode previews - floor-clamped no-visible-change counter advancement Mention any dependencies, configuration changes, or additional requirements introduced. - Configuration change: new optional field `auto_price_reduction.on_update` (default `false`, backward compatible). - No external runtime dependency changes. - Existing configurations continue to work unchanged unless `on_update` is enabled. ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [x] 🐞 Bug fix (non-breaking change which fixes an issue) - [x] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * `verify` now previews automatic price-reduction outcomes for both publish and update modes. * New config option (auto_price_reduction.on_update, default false) to enable reductions during update runs. * **Documentation** * Clarified publish vs. update reduction behavior, timing rules, restore-first semantics, logging, examples, and troubleshooting. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
727366faef |
fix(publish): rewrite shipping dialog selectors for redesigned DOM (#956) (#957)
## ℹ️ Description - Link to the related issue(s): Issue #956 - The shipping dialog on kleinanzeigen.de was completely redesigned, breaking all DOM selectors used by `__set_shipping_options()`. Radio buttons lost their stable IDs (`#radio-button-SMALL` etc.) and checkboxes lost their `data-testid` attributes — both now use React-generated IDs. The fix targets stable `value` attributes instead, which are API-level identifiers used by the site's JS. ## 📋 Changes Summary - Add shared shipping constants (`CARRIER_CODE_BY_OPTION`, `OPTION_NAME_BY_CARRIER_CODE`, `SIZE_INFO_BY_CARRIER_CODE`, `CARRIER_CODES_BY_SIZE`) to `ad_model.py` as single source of truth for both publishing and extraction - Rewrite `__set_shipping_options()` in `__init__.py` with new selectors: - Size radios: `input[@type="radio" and @value="SMALL|MEDIUM|LARGE"]` - Carrier checkboxes: `input[@type="checkbox" and @value="HERMES_001"]` etc. - Use `attrs.get("checked") is not None` consistently (matches existing pattern at line 2540) - Re-raise `TimeoutError` with descriptive translatable message instead of silently swallowing - Update `extract.py` to use shared `OPTION_NAME_BY_CARRIER_CODE` (DRY) - Add German translation for new error message - Update `docs/AD_CONFIGURATION.md` with shipping options reference table - Verified against live site: all 3 size groups (Klein/Mittel/Groß) and individual shipping pass ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [x] 🐞 Bug fix (non-breaking change which fixes an issue) - [ ] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test` — 990 passed, 4 skipped). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint` — 0 errors, 0 warnings). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a Shipping Options Reference table, clarified size-group rules, tightened sell_directly requirements, and updated the example ad to use predefined shipping options. * **Bug Fixes** * Enforce that selected shipping options belong to a single package size. * Missing shipping options now raise an explicit error. * Improved dialog interaction, timeout handling, and reliability when configuring shipping. * **Tests** * Updated dialog tests, added coverage for selection modes and error cases, and removed obsolete async helper tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
8cf40dda30 |
fix(publish): align post-ad selectors with current DOM (#936)
## ℹ️ Description *Provide a concise summary of the changes introduced in this pull request.* - Link to the related issue(s): #920 - Describe the motivation and context for this change. Publishing failed on live ads because the Kleinanzeigen post-ad DOM changed (dotted special-attribute keys, new condition dialog structure, and shipping radios). This PR updates selector logic to the current DOM contract, hardens XPath handling, and adds regression coverage. ## 📋 Changes Summary - Updated publish selector handling in `KleinanzeigenBot`: - WANTED shipping now uses `ad-shipping-enabled-yes/no` radios with quick DOM checks. - `__set_condition()` now opens the new dialog via `.condition` label context and selects by radio `value`. - `__set_special_attributes()` now normalizes keys for current dotted DOM fields, validates selector tokens, safely escapes XPath literals, supports compound `@name` cases, and handles missing `id` attributes via XPath fallback. - `__set_shipping()` removed legacy `.versand_s` assumptions and now uses explicit step-scoped timeout errors for dialog handling. - Added/updated unit tests in `tests/unit/test_init.py` for: - condition dialog selection behavior, - shipping radio behavior (including already-selected paths), - special attribute non-string conversion and compound-name lookup. - Updated docs in `docs/AD_CONFIGURATION.md` to clarify that `condition_s` uses API values from downloaded ads. - Updated German translations for newly introduced/adjusted user-facing error strings. Mention any dependencies, configuration changes, or additional requirements introduced. - No dependency or config schema changes. ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [x] 🐞 Bug fix (non-breaking change which fixes an issue) - [ ] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified condition rules: require API condition values, listed common values, added example mappings and YAML guidance. * **Bug Fixes** * More reliable condition selection; strengthened special-attribute validation and lookup; improved shipping toggle and dialog flows with clearer localized timeout errors and consistent cost input formatting. * **Tests** * Expanded unit tests for condition selection, compound special-attribute resolution, shipping dialog flows, and wanted-ad shipping behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
b143b26c24 |
enh(download): enforce single-use placeholders in naming templates (#909)
## ℹ️ Description This pull request finalizes PR5 by simplifying download naming template rules and making truncation behavior deterministic and reviewable. - Link to the related issue(s): Issue #841 - Describe the motivation and context for this change. Repeated placeholders and implicit truncation priority made naming outcomes hard to predict. This change enforces single-use placeholders and aligns validation, rendering, docs, and tests around explicit priority rules under tight budgets. ## 📋 Changes Summary - Enforced template validation rule: each placeholder (`{id}`, `{title}`) may appear at most once in: - `download.folder_name_template` - `download.ad_file_name_template` - Added validation error path and i18n entry for repeated placeholders. - Refined `_render_download_name_with_budget` behavior to deterministic priority: - `{id}` protected as much as possible - literals preserved after `{id}` - `{title}` truncated first under budget pressure - Added truncation warnings and bounded log previews for template/title/rendered values. - Improved snippet truncation behavior to keep preview length within configured caps, including small/zero limit handling. - Expanded tests in `test_config_model.py` and `test_extract.py` to cover: - repeated placeholder rejection - id/title truncation warnings - helper edge cases for log snippet truncation - Updated configuration documentation with: - valid/invalid template examples - tight-budget examples reflecting current priority behavior - explicit note about last-resort `{id}` truncation - Updated generated artifacts where applicable: - `docs/config.default.yaml` - `schemas/config.schema.json` - No new runtime dependencies or new config keys introduced. ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [x] ✨ New feature (adds new functionality without breaking existing usage) - [x] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. |
||
|
Jens
|
e1bb54ef53 |
enh: replace login UNKNOWN state with boolean + detection reason (#890)
## ℹ️ Description This PR implements issue #872 by replacing the tri-state login model (`LOGGED_IN` / `LOGGED_OUT` / `UNKNOWN`) with an explicit login detection result object that separates control flow from diagnostics. - Link to the related issue(s): Issue #872 - Describe the motivation and context for this change. - `UNKNOWN` previously mixed two concerns: decision logic and observability. In practice, callers mostly needed a binary decision (logged in or not), while diagnostics needed the *reason* for inconclusive detection. - This change makes login decisions explicit (`is_logged_in`) and keeps diagnostics/context explicit (`reason`), reducing ambiguity and making behavior easier to reason about and test. ## 📋 Changes Summary - Replaced `LoginState` with: - `LoginDetectionReason` enum (`USER_INFO_MATCH`, `CTA_MATCH`, `SELECTOR_TIMEOUT`) - `LoginDetectionResult` frozen dataclass (`is_logged_in`, `reason`) with invariant checks in `__post_init__` - Updated `get_login_state()` to return `LoginDetectionResult` while preserving detection order and runtime behavior: 1. DOM logged-in selector match -> `USER_INFO_MATCH` 2. Logged-out CTA selector match -> `CTA_MATCH` 3. Inconclusive fallback -> `SELECTOR_TIMEOUT` (+ optional diagnostics capture) - Updated login flow call sites to branch on `result.is_logged_in` and include reason context in post-login failure handling. - Updated diagnostics text from UNKNOWN-specific wording to inconclusive wording. - Renamed diagnostics artifact prefix from `login_detection_unknown` to `login_detection_inconclusive` and aligned diagnostics utility docs. - Migrated and expanded unit tests: - switched enum-based assertions/mocks to `LoginDetectionResult` - added invariant tests for invalid result combinations - Updated docs to match current behavior and config keys: - removed outdated auth-probe fallback wording - documented `CTA_MATCH` / `SELECTOR_TIMEOUT` meaning and troubleshooting guidance - added migration note from `diagnostics.login_detection_capture` to `diagnostics.capture_on.login_detection` Mention any dependencies, configuration changes, or additional requirements introduced. - No new runtime dependencies introduced. - No user-facing config schema changes introduced. ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [x] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * DOM-first login detection with a logged-out CTA check, explicit inconclusive outcome (SELECTOR_TIMEOUT), and reason-aware detection surfaced in logs/errors. * **Documentation** * Updated troubleshooting and configuration docs and examples to use diagnostics.capture_on.login_detection and added migration guidance for the legacy key. * **Tests** * Updated and added tests covering new detection outcomes, reason validation, and diagnostic capture behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
2abe0ccb71 | feat: add configurable download naming templates (#896) | ||
|
Jens
|
2d02d206ec | feat: add configurable download directory (#894) | ||
|
Jens
|
930b3f6028 | feat: unify pdm test defaults and verbosity controls (#836) | ||
|
Jens
|
4ae46f7aa4 | feat: preview auto price reduction decisions in verify command (#829) | ||
|
Jens
|
398286bcbc |
ci: check generated schema and default config artifacts (#825)
## ℹ️ Description - Link to the related issue(s): N/A - Add a CI guard that fails when generated artifacts are out of sync, motivated by preventing missing schema updates and keeping generated reference files current. - Add a committed `docs/config.default.yaml` as a user-facing default configuration reference. ## 📋 Changes Summary - Add `scripts/check_generated_artifacts.py` to regenerate schema artifacts and compare tracked outputs (`schemas/*.json` and `docs/config.default.yaml`) against generated content. - Run the new artifact consistency check in CI via `.github/workflows/build.yml`. - Add `pdm run generate-config` and `pdm run generate-artifacts` tasks, with a cross-platform-safe delete in `generate-config`. - Add generated `docs/config.default.yaml` and document it in `docs/CONFIGURATION.md`. - Update `schemas/config.schema.json` with the `diagnostics.timing_collection` property generated from the model. ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [x] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a reference link to the default configuration snapshot for easier access to baseline settings. * **Chores** * Added a CI build-time check that validates generated schemas and the default config and alerts when regeneration is needed. * Added scripts to generate the default config and to sequence artifact generation. * Added a utility to produce standardized schema content and compare generated artifacts. * Minor tweak to schema generation success messaging. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
55777710e8 | feat: explain auto price reduction decisions and traces (#826) | ||
|
Jens
|
50fc8781a9 | feat: collect timeout timing sessions for diagnostics (#814) | ||
|
Jens
|
4282b05ff3 | fix: add explicit workspace mode resolution for --config (#818) | ||
|
Jens
|
601b405ded | fix: improve logging messages and documentation (#803) | ||
|
Jens
|
b3d5a4b228 | feat: capture publish failure diagnostics with screenshot and logs (#802) | ||
|
Jens
|
a4946ba104 |
docs: refactor guides for clearer navigation (#795)
## ℹ️ Description Refactors and reorganizes documentation to improve navigation and keep the README concise. - Link to the related issue(s): Issue #N/A - Describe the motivation and context for this change. - The README had grown long and duplicated detailed config/ad references; this consolidates docs into focused guides and adds an index. ## 📋 Changes Summary - Add dedicated docs pages for configuration, ad configuration, update checks, and a docs index. - Slim README and CONTRIBUTING to reference dedicated guides and clean up formatting/markdownlint issues. - Refresh browser troubleshooting and update-check guidance; keep the update channel name aligned with schema/implementation. - Add markdownlint configuration for consistent docs formatting. ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [x] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Reorganized and enhanced contributing guidelines with improved structure and formatting * Streamlined README with better organization and updated installation instructions * Added comprehensive configuration reference documentation for configuration and ad settings * Improved browser troubleshooting guide with updated guidance, examples, and diagnostic information * Created new documentation index for easier navigation <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
b4cb979164 | fix: auth probe + diagnostics for UNKNOWN states (#791) | ||
|
Jens
|
25079c32c0 |
fix: increase login detection timeout to fix intermittent failures (#701) (#726)
## ℹ️ Description This PR fixes intermittent login detection failures where the bot fails to detect existing login sessions and unnecessarily re-logins, potentially causing IP blocks. - Link to the related issue(s): Issue #701 - Describe the motivation and context for this change: Users reported that the bot sometimes fails to detect existing login sessions (50/50 behavior), especially for browser profiles that haven't been used for 20+ days. This appears to be a race condition where: 1. `web_open()` completes when `document.readyState == 'complete'` 2. But kleinanzeigen.de's client-side JavaScript hasn't yet rendered user profile elements 3. The login detection timeout (5s default) is too short for slow networks or sessions requiring server-side validation ## 📋 Changes Summary - **Add dedicated `login_detection` timeout** to `TimeoutConfig` (default: 10s, previously used generic 5s timeout) - **Apply timeout to both DOM checks** in `is_logged_in()`: `.mr-medium` and `#user-email` elements - **Add debug logging** to track which element detected login or if no login was found - **Regenerate JSON schema** to include new timeout configuration - **Effective total timeout**: ~22.5s (10s base × 1.0 multiplier × 1.5 backoff × 2 retries) vs previous ~11.25s ### Benefits: - Addresses race condition between page load completion and client-side rendering - Provides sufficient time for sessions requiring server-side validation (20+ days old) - User-configurable via `timeouts.login_detection` in `config.yaml` - Follows established pattern of dedicated timeouts (`sms_verification`, `gdpr_prompt`, etc.) ### ⚙️ Type of Change - [x] 🐞 Bug fix (non-breaking change which fixes an issue) - [ ] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a configurable login-detection timeout (default 10s, min 1s) to tune session detection. * **Bug Fixes** * More reliable login checks using a timeout-aware, two-step detection sequence. * Improved diagnostic logging for login attempts, retry behavior, detection outcomes, and timeout events. * **Documentation** * Added troubleshooting guidance explaining the login-detection timeout and when to adjust it. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
89df56bf8b |
test: strengthen coverage for sessions, logging, and update check (#686)
## ℹ️ Description * Strengthen the session/logging/update-check tests to exercise real resources and guards while bringing the update-check docs in line with the supported interval units. - Link to the related issue(s): Issue #N/A ## 📋 Changes Summary - Reworked the `WebScrapingMixin` session tests so they capture each `stop` handler before the browser reference is nulled, ensuring cleanup logic is exercised without crashing. - Added targeted publish and update-check tests that patch the async helpers, guard logic, and logging handlers while confirming `requests.get` is skipped when the state gate is closed. - Updated `docs/update-check.md` to list only the actually supported interval units (up to 30 days) and noted the new guard coverage in the changelog. ### ⚙️ Type of Change - [x] 🐞 Bug fix (non-breaking change which fixes an issue) - [ ] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded test coverage for publish workflow orchestration and update checking interval behavior. * Added comprehensive browser session cleanup tests, including idempotent operations and edge case handling. * Consolidated logging configuration tests with improved handler management validation. * Refined test fixtures and assertions for better test reliability. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
a3ac27c441 |
feat: add configurable timeouts (#673)
## ℹ️ Description - Related issues: #671, #658 - Introduces configurable timeout controls plus retry/backoff handling for flaky DOM operations. We often see timeouts which are note reproducible in certain configurations. I suspect timeout issues based on a combination of internet speed, browser, os, age of the computer and the weather. This PR introduces a comprehensive config model to tweak timeouts. ## 📋 Changes Summary - add TimeoutConfig to the main config/schema and expose timeouts in README/docs - wire WebScrapingMixin, extractor, update checker, and browser diagnostics to honor the configurable timeouts and retries - update translations/tests to cover the new behaviour and ensure lint/mypy/pyright pipelines remain green ### ⚙️ Type of Change - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [x] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Centralized, configurable timeout system for web interactions, detection flows, publishing, and pagination. * Optional retry with exponential backoff for operations that time out. * **Improvements** * Replaced fixed wait times with dynamic timeouts throughout workflows. * More informative timeout-related messages and diagnostics. * **Tests** * New and expanded test coverage for timeout behavior, pagination, diagnostics, and retry logic. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens
|
33d1964f86 |
feat: speed up and stabilise test suite (#676)
## ℹ️ Description *Provide a concise summary of the changes introduced in this pull request.* - Link to the related issue(s): Issue # - Describe the motivation and context for this change. Refactors the test harness for faster and more reliable feedback: adds deterministic time freezing for update checks, accelerates and refactors smoke tests to run in-process, defaults pytest to xdist with durations tracking, and adjusts CI triggers so PRs run the test matrix only once. ## 📋 Changes Summary - add pytest-xdist + durations reporting defaults, force deterministic locale and slow markers, and document the workflow adjustments - run smoke tests in-process (no subprocess churn), mock update checks/logging, and mark slow specs appropriately - deflake update check interval tests by freezing datetime and simplify FixedDateTime helper - limit GitHub Actions `push` trigger to `main` so feature branches rely on the single pull_request run ### ⚙️ Type of Change Select the type(s) of change(s) included in this pull request: - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [x] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist Before requesting a review, confirm the following: - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Ensure tests run in a consistent English locale and restore prior locale after each run * Mark integration scraping tests as slow for clearer categorization * Replace subprocess-based CLI tests with an in-process runner that returns structured results and captures combined stdout/stderr/logs; disable update checks during smoke tests * Freeze current time in update-check tests for deterministic assertions * Add mock for process enumeration in web‑scraping unit tests to stabilize macOS-specific warnings <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Jens Bergmann
|
332926519d | feat: chrome version detection clean (#607) | ||
|
Jens Bergmann
|
c9d04da70d | feat: browser connection improvements (#601) | ||
|
Jens Bergmann
|
1a1633e12d | feat: introduce smoke test group and fail-fast test orchestration (#572) | ||
|
Jens Bergmann
|
5430f5cdc6 |
feat: update check (#561)
feat(update-check): add robust update check with interval support, state management, and CLI integration - Implement version and interval-based update checks with configurable settings - Add CLI command `kleinanzeigen-bot update-check` for manual checks - Introduce state file with versioning, UTC timestamps, and migration logic - Validate and normalize intervals (1d–4w) with fallback for invalid values - Ensure correct handling of timezones and elapsed checks - Improve error handling, logging, and internationalization (i18n) - Add comprehensive test coverage for config, interval logic, migration, and CLI - Align default config, translations, and schema with new functionality - Improve help command UX by avoiding config/log loading for `--help` - Update documentation and README with full feature overview |
