Commit Graph

6 Commits

Author SHA1 Message Date
Jens
eda1b4d0ec feat: add browser profile XDG support and documentation (#777) 2026-01-23 22:45:22 +01:00
Jens
33d1964f86 feat: speed up and stabilise test suite (#676)
## ℹ️ Description
*Provide a concise summary of the changes introduced in this pull
request.*

- Link to the related issue(s): Issue #
- Describe the motivation and context for this change.

Refactors the test harness for faster and more reliable feedback: adds
deterministic time freezing for update checks, accelerates and refactors
smoke tests to run in-process, defaults pytest to xdist with durations
tracking, and adjusts CI triggers so PRs run the test matrix only once.

## 📋 Changes Summary

- add pytest-xdist + durations reporting defaults, force deterministic
locale and slow markers, and document the workflow adjustments
- run smoke tests in-process (no subprocess churn), mock update
checks/logging, and mark slow specs appropriately
- deflake update check interval tests by freezing datetime and simplify
FixedDateTime helper
- limit GitHub Actions `push` trigger to `main` so feature branches rely
on the single pull_request run

### ⚙️ Type of Change
Select the type(s) of change(s) included in this pull request:
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [x]  New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)


##  Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Ensure tests run in a consistent English locale and restore prior
locale after each run
  * Mark integration scraping tests as slow for clearer categorization
* Replace subprocess-based CLI tests with an in-process runner that
returns structured results and captures combined stdout/stderr/logs;
disable update checks during smoke tests
* Freeze current time in update-check tests for deterministic assertions
* Add mock for process enumeration in web‑scraping unit tests to
stabilize macOS-specific warnings

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-12 21:29:51 +01:00
Jens
339d66ed47 feat: Replace custom RemoteObject wrapper with direct NoDriver 0.47+ usage (#652)
## ℹ️ Description
*Replace custom RemoteObject serialization wrapper with direct NoDriver
0.47+ RemoteObject API usage for better performance and
maintainability.*

- **Motivation**: The custom wrapper was unnecessary complexity when
NoDriver 0.47+ provides direct RemoteObject API
- **Context**: Upgrading from NoDriver 0.39 to 0.47 introduced
RemoteObject, and we want to use it as intended
- **Goal**: Future-proof implementation using the standard NoDriver
patterns

## 📋 Changes Summary

- Replace custom serialization wrapper with direct RemoteObject API
usage
- Implement proper RemoteObject detection and conversion in
web_execute()
- Add comprehensive _convert_remote_object_value() method for recursive
conversion
- Handle key/value list format from deep_serialized_value.value
- Add type guards and proper type checking for RemoteObject instances
- Maintain internal API stability while using RemoteObject as intended
- Add 19 comprehensive test cases covering all conversion scenarios
- Application tested and working with real ad download, update and
publish

### ⚙️ Type of Change
- [x]  New feature (adds new functionality without breaking existing
usage)
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)

##  Checklist
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (pdm run
test).
- [x] I have formatted the code (pdm run format).
- [x] I have verified that linting passes (pdm run lint).
- [x] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
2025-10-20 08:52:06 +02:00
Jens
7b4b7907d0 feat: cleanup test structure and remove BelenConf testing (#639) 2025-10-14 09:50:50 +02:00
Jens
36ca178574 feat: upgrade nodriver from 0.39 to 0.47 (#635)
## ℹ️ Description
Upgrade nodriver dependency from pinned version 0.39.0 to latest 0.47.0
to resolve browser startup issues and JavaScript evaluation problems
that affected versions 0.40-0.44.

- Link to the related issue(s): Resolves nodriver compatibility issues
- This upgrade addresses browser startup problems and window.BelenConf
evaluation failures that were blocking the use of newer nodriver
versions.

## 📋 Changes Summary

- Updated nodriver dependency from pinned 0.39.0 to >=0.47.0 in
pyproject.toml
- Fixed RemoteObject handling in web_execute method for nodriver 0.47
compatibility
- Added comprehensive BelenConf test fixture with real production data
structure
- Added integration test to validate window.BelenConf evaluation works
correctly
- Added German translation for new error message
- Replaced real user data with privacy-safe dummy data in test fixtures

### 🔧 Type Safety Improvements

**Added explicit `str()` conversions to resolve type inference issues:**

The comprehensive BelenConf test fixture contains deeply nested data
structures that caused pyright's type checker to infer complex
dictionary types throughout the codebase. To ensure type safety and
prevent runtime errors, I added explicit `str()` conversions in key
locations:

- **CSRF tokens**: `str(csrf_token)` - Ensures CSRF tokens are treated
as strings
- **Special attributes**: `str(special_attribute_value)` - Converts
special attribute values to strings
- **DOM attributes**: `str(special_attr_elem.attrs.id)` - Ensures
element IDs are strings
- **URL handling**: `str(current_img_url)` and `str(href_attributes)` -
Converts URLs and href attributes to strings
- **Price values**: `str(ad_cfg.price)` - Ensures price values are
strings

These conversions are defensive programming measures that ensure
backward compatibility and prevent type-related runtime errors, even if
the underlying data structures change in the future.

### ⚙️ Type of Change
- [x]  New feature (adds new functionality without breaking existing
usage)
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)

##  Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
2025-10-12 21:22:46 +02:00
Jens Bergmann
50656ad7e2 feat: Improve test coverage (#515)
* test: implement comprehensive test coverage improvements

This commit improves test coverage across multiple modules, adding unit tests
for core functionality.

Key improvements:

1. WebScrapingMixin:
   - Add comprehensive async error handling tests
   - Add session management tests (browser crash recovery, session expiration)
   - Add element interaction tests (custom wait conditions, timeouts)
   - Add browser configuration tests (extensions, preferences)
   - Add robust awaitable mocking infrastructure
   - Rename integration test file to avoid naming conflicts

2. Error Handlers:
   - Add tests for error message formatting
   - Add tests for error recovery scenarios
   - Add tests for error logging functionality

3. Network Utilities:
   - Add tests for port checking functionality
   - Add tests for network error handling
   - Add tests for connection management

4. Pydantic Models:
   - Add tests for validation cases
   - Add tests for error handling
   - Add tests for complex validation scenarios

Technical details:
- Use TrulyAwaitableMockPage for proper async testing
- Add comprehensive mocking for browser and page objects
- Add proper cleanup in session management tests
- Add browser-specific configuration tests (Chrome/Edge)
- Add proper type hints and docstrings

Files changed:
- Renamed: tests/integration/test_web_scraping_mixin.py → tests/integration/test_web_scraping_mixin_integration.py
- Added: tests/unit/test_error_handlers.py
- Added: tests/unit/test_net.py
- Added: tests/unit/test_pydantics.py
- Added: tests/unit/test_web_scraping_mixin.py

* test: enhance test coverage with additional edge cases and scenarios

This commit extends the test coverage improvements with additional test cases
and edge case handling, focusing on browser configuration, error handling, and
file utilities.

Key improvements:

1. WebScrapingMixin:
   - Add comprehensive browser binary location detection tests
   - Add cross-platform browser path detection (Linux, macOS, Windows)
   - Add browser profile configuration tests
   - Add session state persistence tests
   - Add external process termination handling
   - Add session creation error cleanup tests
   - Improve browser argument configuration tests
   - Add extension loading validation tests

2. Error Handlers:
   - Add debug mode error handling tests
   - Add specific error type tests (AttributeError, ImportError, NameError, TypeError)
   - Improve error message formatting tests
   - Add traceback inclusion verification

3. Pydantic Models:
   - Add comprehensive validation error message tests
   - Add tests for various error codes and contexts
   - Add tests for pluralization in error messages
   - Add tests for empty error list handling
   - Add tests for context handling in validation errors

4. File Utilities:
   - Add comprehensive path resolution tests
   - Add tests for file and directory reference handling
   - Add tests for special path cases
   - Add tests for nonexistent path handling
   - Add tests for absolute and relative path conversion

Technical details:
- Add proper type casting for test fixtures
- Improve test isolation and cleanup
- Add platform-specific browser path detection
- Add proper error context handling
- Add comprehensive error message formatting tests
- Add proper cleanup in session management tests
- Add browser-specific configuration tests
- Add proper path normalization and resolution tests

* fix(test): handle Linux browser paths in web_scraping_mixin test

Update mock_exists to properly detect Linux browser binaries in test_browser_profile_configuration, fixing the "Installed browser could not be detected" error.

* fix(test): handle Windows browser paths in web_scraping_mixin test

Add Windows browser paths to mock_exists function to properly detect browser binaries on Windows platform, fixing the "Specified browser binary does not exist" error.
2025-05-18 19:02:59 +02:00