## ℹ️ Description
*Provide a concise summary of the changes introduced in this pull
request.*
- Link to the related issue(s): n/a
- Describe the motivation and context for this change.
This replaces the stacked XPath work from #845 with a standalone fix
from `main`. It makes `By.XPATH` use the native page XPath API instead
of routing XPath selectors through text lookup.
## 📋 Changes Summary
- Add private XPath helpers in `WebScrapingMixin` for first-match and
all-match lookups.
- Route `By.XPATH` in `_web_find_once()` and `_web_find_all_once()`
through `page.xpath(...)`.
- Add unit coverage for XPath helper behavior, empty results, and
unsupported parent scoping.
- No configuration changes or new dependencies.
### ⚙️ Type of Change
Select the type(s) of change(s) included in this pull request:
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ] ✨ New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)
## ✅ Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactoring**
* Improved web scraping element selection reliability through
streamlined XPath operations and better internal helper methods.
* **Tests**
* Added comprehensive unit tests for XPath-based element lookup
operations to ensure consistent behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## ℹ️ Description
This pull request fixes Windows browser auto-detection failures reported
by users where `diagnose`/startup could not find an installed browser
even when Chrome or Edge were present in standard locations. It also
makes diagnostics resilient when auto-detection fails by avoiding an
assertion-driven abort and continuing with a clear failure log.
- Link to the related issue(s): Issue #815
- Describe the motivation and context for this change.
- Users reported `Installed browser could not be detected` on Windows
despite having a browser installed.
- The previous Windows candidate list used a mix of incomplete paths and
direct `os.environ[...]` lookups that could raise when variables were
missing.
- The updated path candidates and ordering were aligned with common
Windows install locations used by Playwright’s channel/executable
resolution logic (Chrome/Edge under `LOCALAPPDATA`, `PROGRAMFILES`, and
`PROGRAMFILES(X86)`).
## 📋 Changes Summary
- Expanded Windows browser path candidates in `get_compatible_browser()`
to include common Google Chrome and Microsoft Edge install paths, while
keeping Chromium and PATH fallbacks.
- Replaced unsafe direct env-var indexing with safe retrieval
(`os.environ.get(...)`) and added a fallback derivation for
`LOCALAPPDATA` via `USERPROFILE\\AppData\\Local` when needed.
- Kept legacy Chrome path candidates
(`...\\Chrome\\Application\\chrome.exe`) as compatibility fallback.
- Updated diagnostics flow to catch browser auto-detection assertion
failures and continue with `(fail) No compatible browser found` instead
of crashing.
- Added/updated unit tests to verify:
- Windows detection for LocalAppData Chrome/Edge/Chromium paths.
- Missing Windows env vars no longer cause key lookup failures and still
surface the intended final detection assertion.
- `diagnose_browser_issues()` handles auto-detection assertion failures
without raising and logs the expected failure message.
### ⚙️ Type of Change
Select the type(s) of change(s) included in this pull request:
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)
## ✅ Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Hardened Windows browser auto-detection: checks additional common
installation locations for Chrome/Chromium/Edge and treats detection
failures as non-fatal, allowing diagnostics to continue with fallback
behavior and debug logging when no browser is found.
* **Tests**
* Expanded Windows detection tests to cover more path scenarios and
added cases verifying failure-mode diagnostics and logging.
* **Style**
* Minor formatting tweak in default configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## ℹ️ Description
Added Webselect-Function for Input/Dropdown Combobox
PR for issue/missing feature #677
# Fixes / Enhancements
Finding Special Attributes Elements can fail because they are currently
only selected using the name="..." attributes of the HTML elements. If
it fails, ALSO fallback-handle selecting special attribute HTML elements
by ID instead / additionally. (For example the "brands" Input/Combobox
for Mens Shoes...
When trying to select a Value in a <select>, it does not only rely on
the actual Option value (xxx in the example <options
value="xxx">yyy</...>) but instead also on the displayed HTML value
(i.e. yyy in above example). This improves UX because the User doesnt
have to check the actual "value" of the Option but instead can check the
displayed Value from the Browsers Display directly.
Testcases for Webselect_Combobox were not added due to missing knowledge
about Async Mocking properly.
## 📋 Changes Summary
✅ Fixes & Enhancements
- New WebSelect Functionality
- Improved Element Detection for Special Attributes
- Enhanced <select> Option Matching Logic
This improves UX and test robustness — users no longer need to know the
exact underlying value, as matching also works with the visible label
shown in the browser.
🧩 Result
These updates make dropdown and combobox interactions more intuitive,
resilient, and user-friendly across diverse HTML structures.
### ⚙️ Type of Change
Select the type(s) of change(s) included in this pull request:
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)
- [x] ✨ New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)
## ✅ Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [ ] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Field lookup now falls back to locating by ID when name lookup times
out.
* Option selection uses a two-pass match (value then displayed text);
JS-path failures now surface as timeouts.
* Error and log messages localized and clarified.
* **New Features**
* Support for combobox-style inputs: type into the input, open dropdown,
and select by visible text (handles special characters).
* **Tests**
* Added tests for combobox selection, missing dropdowns, no-match
errors, value-path selection, and special-character handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Jens <1742418+1cu@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
## ℹ️ Description
Eliminate all blocking I/O operations in async contexts and modernize
file path handling by migrating from os.path to pathlib.Path.
- Link to the related issue(s): #692
- Get rid of the TODO in pyproject.toml
- The added debug logging will ease the troubleshooting for path related
issues.
## 📋 Changes Summary
- Enable ASYNC210, ASYNC230, ASYNC240, ASYNC250 Ruff rules
- Wrap blocking urllib.request.urlopen() in run_in_executor
- Wrap blocking file operations (open, write) in run_in_executor
- Replace blocking os.path calls with async helpers using
run_in_executor
- Replace blocking input() with await ainput()
- Migrate extract.py from os.path to pathlib.Path
- Use Path() constructor and / operator for path joining
- Use Path.mkdir(), Path.rename() in executor instead of os functions
- Create mockable _path_exists() and _path_is_dir() helpers
- Add debug logging for all file system operations
### ⚙️ Type of Change
Select the type(s) of change(s) included in this pull request:
- [X] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ] ✨ New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)
## ✅ Checklist
Before requesting a review, confirm the following:
- [X] I have reviewed my changes to ensure they meet the project's
standards.
- [X] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [X] I have formatted the code (`pdm run format`).
- [X] I have verified that linting passes (`pdm run lint`).
- [X] I have updated documentation where necessary.
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Made user prompt non‑blocking to improve responsiveness.
* Converted filesystem/path handling and prefs I/O to async‑friendly
operations; moved blocking network and file work to background tasks.
* Added async file/path helpers and async port‑check before browser
connections.
* **Tests**
* Expanded unit tests for path helpers, image download success/failure,
prefs writing, and directory creation/renaming workflows.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## ℹ️ Description
- Related issues: #671, #658
- Introduces configurable timeout controls plus retry/backoff handling
for flaky DOM operations.
We often see timeouts which are note reproducible in certain
configurations. I suspect timeout issues based on a combination of
internet speed, browser, os, age of the computer and the weather.
This PR introduces a comprehensive config model to tweak timeouts.
## 📋 Changes Summary
- add TimeoutConfig to the main config/schema and expose timeouts in
README/docs
- wire WebScrapingMixin, extractor, update checker, and browser
diagnostics to honor the configurable timeouts and retries
- update translations/tests to cover the new behaviour and ensure
lint/mypy/pyright pipelines remain green
### ⚙️ Type of Change
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [x] ✨ New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)
## ✅ Checklist
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Centralized, configurable timeout system for web interactions,
detection flows, publishing, and pagination.
* Optional retry with exponential backoff for operations that time out.
* **Improvements**
* Replaced fixed wait times with dynamic timeouts throughout workflows.
* More informative timeout-related messages and diagnostics.
* **Tests**
* New and expanded test coverage for timeout behavior, pagination,
diagnostics, and retry logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## ℹ️ Description
*Replace custom RemoteObject serialization wrapper with direct NoDriver
0.47+ RemoteObject API usage for better performance and
maintainability.*
- **Motivation**: The custom wrapper was unnecessary complexity when
NoDriver 0.47+ provides direct RemoteObject API
- **Context**: Upgrading from NoDriver 0.39 to 0.47 introduced
RemoteObject, and we want to use it as intended
- **Goal**: Future-proof implementation using the standard NoDriver
patterns
## 📋 Changes Summary
- Replace custom serialization wrapper with direct RemoteObject API
usage
- Implement proper RemoteObject detection and conversion in
web_execute()
- Add comprehensive _convert_remote_object_value() method for recursive
conversion
- Handle key/value list format from deep_serialized_value.value
- Add type guards and proper type checking for RemoteObject instances
- Maintain internal API stability while using RemoteObject as intended
- Add 19 comprehensive test cases covering all conversion scenarios
- Application tested and working with real ad download, update and
publish
### ⚙️ Type of Change
- [x] ✨ New feature (adds new functionality without breaking existing
usage)
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)
## ✅ Checklist
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (pdm run
test).
- [x] I have formatted the code (pdm run format).
- [x] I have verified that linting passes (pdm run lint).
- [x] I have updated documentation where necessary.
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
## ℹ️ Description
*Fixes the nodriver 0.47.0 RemoteObject conversion bug that was causing
KeyError and TypeError when accessing BelenConf dimensions.*
- Link to the related issue(s): Issue #650
- The bot was crashing when downloading ads because nodriver 0.47.0 was
returning JavaScript objects as lists of [key, value] pairs instead of
proper Python dictionaries, causing BelenConf dimensions to be
inaccessible.
## 📋 Changes Summary
- **Fixed nodriver RemoteObject conversion bug** in
`web_scraping_mixin.py`:
- Added detection logic for list-of-pairs format in `web_execute` method
- Enhanced `_convert_remote_object_dict` to recursively convert nested
structures
- Now properly converts JavaScript objects to Python dictionaries
- **Bot functionality fully restored** - can now download ads with
subcategories and special attributes
### ⚙️ Type of Change
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ] ✨ New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)
## ✅ Checklist
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
## ℹ️ Description
*Provide a concise summary of the changes introduced in this pull
request.*
- Link to the related issue(s): #644
- Describe the motivation and context for this change.
This PR resolves compatibility issues with nodriver 0.47+ where
page.evaluate() returns RemoteObject instances that need special
handling for proper conversion to Python objects. The update introduced
breaking changes in how JavaScript evaluation results are returned,
causing TypeError: [RemoteObject] object is not subscriptable errors.
## 📋 Changes Summary
- Fixed TypeError: [RemoteObject] object is not subscriptable in
web_request() method
- Added comprehensive RemoteObject conversion logic with
_convert_remote_object_result()
- Added _convert_remote_object_dict() for recursive nested structure
conversion
- Fixed price field concatenation issue in MODIFY mode by explicit field
clearing
- Updated web_sleep() to accept integer milliseconds instead of float
seconds
- Updated German translations for new log messages
- Fixed linting issues (E711, E712) in test assertions
### ⚙️ Type of Change
Select the type(s) of change(s) included in this pull request:
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ] ✨ New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)
## ✅ Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (pdm run
test).
- [x] I have formatted the code (pdm run format).
- [x] I have verified that linting passes (pdm run lint).
- [x] I have updated documentation where necessary.
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
## ℹ️ Description
Upgrade nodriver dependency from pinned version 0.39.0 to latest 0.47.0
to resolve browser startup issues and JavaScript evaluation problems
that affected versions 0.40-0.44.
- Link to the related issue(s): Resolves nodriver compatibility issues
- This upgrade addresses browser startup problems and window.BelenConf
evaluation failures that were blocking the use of newer nodriver
versions.
## 📋 Changes Summary
- Updated nodriver dependency from pinned 0.39.0 to >=0.47.0 in
pyproject.toml
- Fixed RemoteObject handling in web_execute method for nodriver 0.47
compatibility
- Added comprehensive BelenConf test fixture with real production data
structure
- Added integration test to validate window.BelenConf evaluation works
correctly
- Added German translation for new error message
- Replaced real user data with privacy-safe dummy data in test fixtures
### 🔧 Type Safety Improvements
**Added explicit `str()` conversions to resolve type inference issues:**
The comprehensive BelenConf test fixture contains deeply nested data
structures that caused pyright's type checker to infer complex
dictionary types throughout the codebase. To ensure type safety and
prevent runtime errors, I added explicit `str()` conversions in key
locations:
- **CSRF tokens**: `str(csrf_token)` - Ensures CSRF tokens are treated
as strings
- **Special attributes**: `str(special_attribute_value)` - Converts
special attribute values to strings
- **DOM attributes**: `str(special_attr_elem.attrs.id)` - Ensures
element IDs are strings
- **URL handling**: `str(current_img_url)` and `str(href_attributes)` -
Converts URLs and href attributes to strings
- **Price values**: `str(ad_cfg.price)` - Ensures price values are
strings
These conversions are defensive programming measures that ensure
backward compatibility and prevent type-related runtime errors, even if
the underlying data structures change in the future.
### ⚙️ Type of Change
- [x] ✨ New feature (adds new functionality without breaking existing
usage)
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)
## ✅ Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
