mirror of
https://github.com/Second-Hand-Friends/kleinanzeigen-bot.git
synced 2026-03-12 02:31:45 +01:00
feat: add configurable timeouts (#673)
## ℹ️ Description - Related issues: #671, #658 - Introduces configurable timeout controls plus retry/backoff handling for flaky DOM operations. We often see timeouts which are note reproducible in certain configurations. I suspect timeout issues based on a combination of internet speed, browser, os, age of the computer and the weather. This PR introduces a comprehensive config model to tweak timeouts. ## 📋 Changes Summary - add TimeoutConfig to the main config/schema and expose timeouts in README/docs - wire WebScrapingMixin, extractor, update checker, and browser diagnostics to honor the configurable timeouts and retries - update translations/tests to cover the new behaviour and ensure lint/mypy/pyright pipelines remain green ### ⚙️ Type of Change - [ ] 🐞 Bug fix (non-breaking change which fixes an issue) - [x] ✨ New feature (adds new functionality without breaking existing usage) - [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations) ## ✅ Checklist - [x] I have reviewed my changes to ensure they meet the project's standards. - [x] I have tested my changes and ensured that all tests pass (`pdm run test`). - [x] I have formatted the code (`pdm run format`). - [x] I have verified that linting passes (`pdm run lint`). - [x] I have updated documentation where necessary. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Centralized, configurable timeout system for web interactions, detection flows, publishing, and pagination. * Optional retry with exponential backoff for operations that time out. * **Improvements** * Replaced fixed wait times with dynamic timeouts throughout workflows. * More informative timeout-related messages and diagnostics. * **Tests** * New and expanded test coverage for timeout behavior, pagination, diagnostics, and retry logic. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
23
README.md
23
README.md
@@ -277,6 +277,27 @@ categories:
|
||||
Verschenken & Tauschen > Verleihen: 272/274
|
||||
Verschenken & Tauschen > Verschenken: 272/192
|
||||
|
||||
# timeout tuning (optional)
|
||||
timeouts:
|
||||
multiplier: 1.0 # Scale all timeouts (e.g. 2.0 for slower networks)
|
||||
default: 5.0 # Base timeout for web_find/web_click/etc.
|
||||
page_load: 15.0 # Timeout for web_open page loads
|
||||
captcha_detection: 2.0 # Timeout for captcha iframe detection
|
||||
sms_verification: 4.0 # Timeout for SMS verification banners
|
||||
gdpr_prompt: 10.0 # Timeout when handling GDPR dialogs
|
||||
publishing_result: 300.0 # Timeout for publishing status checks
|
||||
publishing_confirmation: 20.0 # Timeout for publish confirmation redirect
|
||||
pagination_initial: 10.0 # Timeout for first pagination lookup
|
||||
pagination_follow_up: 5.0 # Timeout for subsequent pagination clicks
|
||||
quick_dom: 2.0 # Generic short DOM timeout (shipping dialogs, etc.)
|
||||
update_check: 10.0 # Timeout for GitHub update requests
|
||||
chrome_remote_probe: 2.0 # Timeout for local remote-debugging probes
|
||||
chrome_remote_debugging: 5.0 # Timeout for remote debugging API calls
|
||||
chrome_binary_detection: 10.0 # Timeout for chrome --version subprocess
|
||||
retry_enabled: true # Enables DOM retry/backoff when timeouts occur
|
||||
retry_max_attempts: 2
|
||||
retry_backoff_factor: 1.5
|
||||
|
||||
# download configuration
|
||||
download:
|
||||
include_all_matching_shipping_options: false # if true, all shipping options matching the package size will be included
|
||||
@@ -329,6 +350,8 @@ login:
|
||||
password: ""
|
||||
```
|
||||
|
||||
Slow networks or sluggish remote browsers often just need a higher `timeouts.multiplier`, while truly problematic selectors can get explicit values directly under `timeouts`. Remember to regenerate the schemas after changing the configuration model so editors stay in sync.
|
||||
|
||||
### <a name="ad-config"></a>2) Ad configuration
|
||||
|
||||
Each ad is described in a separate JSON or YAML file with prefix `ad_<filename>`. The prefix is configurable in config file.
|
||||
|
||||
Reference in New Issue
Block a user