Commit Graph

97 Commits

Author SHA1 Message Date
Jens
220c01f257 fix: eliminate async safety violations and migrate to pathlib (#697)
## ℹ️ Description
Eliminate all blocking I/O operations in async contexts and modernize
file path handling by migrating from os.path to pathlib.Path.

- Link to the related issue(s): #692 
- Get rid of the TODO in pyproject.toml
- The added debug logging will ease the troubleshooting for path related
issues.

## 📋 Changes Summary

- Enable ASYNC210, ASYNC230, ASYNC240, ASYNC250 Ruff rules
- Wrap blocking urllib.request.urlopen() in run_in_executor
- Wrap blocking file operations (open, write) in run_in_executor
- Replace blocking os.path calls with async helpers using
run_in_executor
- Replace blocking input() with await ainput()
- Migrate extract.py from os.path to pathlib.Path
- Use Path() constructor and / operator for path joining
- Use Path.mkdir(), Path.rename() in executor instead of os functions
- Create mockable _path_exists() and _path_is_dir() helpers
- Add debug logging for all file system operations

### ⚙️ Type of Change
Select the type(s) of change(s) included in this pull request:
- [X] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ]  New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)


##  Checklist
Before requesting a review, confirm the following:
- [X] I have reviewed my changes to ensure they meet the project's
standards.
- [X] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [X] I have formatted the code (`pdm run format`).
- [X] I have verified that linting passes (`pdm run lint`).
- [X] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Refactor**
  * Made user prompt non‑blocking to improve responsiveness.
* Converted filesystem/path handling and prefs I/O to async‑friendly
operations; moved blocking network and file work to background tasks.
* Added async file/path helpers and async port‑check before browser
connections.

* **Tests**
* Expanded unit tests for path helpers, image download success/failure,
prefs writing, and directory creation/renaming workflows.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-05 20:53:40 +01:00
Jens
4870bc223f chore: improve coverage reporting (#683)
## ℹ️ Description
* Restrict coverage reporting to library files and collect per-suite
coverage data for Codecov’s flags.
- Link to the related issue(s): Issue #N/A
- Describe the motivation and context for this change.

## 📋 Changes Summary
- add `coverage:prepare` and per-suite `COVERAGE_FILE`s so each test
group writes its own sqlite and XML artifacts without appending
- replace the shell scripts with `scripts/coverage_helper.py`, scope the
report to `src/kleinanzeigen_bot/*`, and add logging/validation around
cleanup, pytest runs, and data combining
- ensure the helper works in CI (accepts extra pytest args, validates
file presence)

### ⚙️ Type of Change
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ]  New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)

##  Checklist
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.
2025-11-16 17:46:02 +01:00
Jens
33d1964f86 feat: speed up and stabilise test suite (#676)
## ℹ️ Description
*Provide a concise summary of the changes introduced in this pull
request.*

- Link to the related issue(s): Issue #
- Describe the motivation and context for this change.

Refactors the test harness for faster and more reliable feedback: adds
deterministic time freezing for update checks, accelerates and refactors
smoke tests to run in-process, defaults pytest to xdist with durations
tracking, and adjusts CI triggers so PRs run the test matrix only once.

## 📋 Changes Summary

- add pytest-xdist + durations reporting defaults, force deterministic
locale and slow markers, and document the workflow adjustments
- run smoke tests in-process (no subprocess churn), mock update
checks/logging, and mark slow specs appropriately
- deflake update check interval tests by freezing datetime and simplify
FixedDateTime helper
- limit GitHub Actions `push` trigger to `main` so feature branches rely
on the single pull_request run

### ⚙️ Type of Change
Select the type(s) of change(s) included in this pull request:
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [x]  New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)


##  Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Ensure tests run in a consistent English locale and restore prior
locale after each run
  * Mark integration scraping tests as slow for clearer categorization
* Replace subprocess-based CLI tests with an in-process runner that
returns structured results and captures combined stdout/stderr/logs;
disable update checks during smoke tests
* Freeze current time in update-check tests for deterministic assertions
* Add mock for process enumeration in web‑scraping unit tests to
stabilize macOS-specific warnings

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-12 21:29:51 +01:00
Jens
71feedc700 fix: pin nodriver to 0.47 (#675)
## ℹ️ Description
- Link to the related issue(s): Issue #N/A
- Describe the motivation and context for this change.

Pin `nodriver` to the last known good 0.47 series so we can avoid the
UTF-8 decoding regression in 0.48.x that currently breaks our local
mypy/linting runs.

## 📋 Changes Summary
- lock runtime dependency `nodriver` to `0.47.*` with an inline comment
describing the upstream regression
- refresh `pdm.lock` so local/CI installs stay on the pinned version

### ⚙️ Type of Change
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ]  New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)


##  Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [ ] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [ ] I have formatted the code (`pdm run format`).
- [ ] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
2025-11-10 11:36:44 +01:00
Sebastian Thomschke
dadd08aedb build: upgrade to Python 3.14 (#636)
Co-authored-by: Jens <1742418+1cu@users.noreply.github.com>
2025-10-14 15:56:35 +02:00
Jens
36ca178574 feat: upgrade nodriver from 0.39 to 0.47 (#635)
## ℹ️ Description
Upgrade nodriver dependency from pinned version 0.39.0 to latest 0.47.0
to resolve browser startup issues and JavaScript evaluation problems
that affected versions 0.40-0.44.

- Link to the related issue(s): Resolves nodriver compatibility issues
- This upgrade addresses browser startup problems and window.BelenConf
evaluation failures that were blocking the use of newer nodriver
versions.

## 📋 Changes Summary

- Updated nodriver dependency from pinned 0.39.0 to >=0.47.0 in
pyproject.toml
- Fixed RemoteObject handling in web_execute method for nodriver 0.47
compatibility
- Added comprehensive BelenConf test fixture with real production data
structure
- Added integration test to validate window.BelenConf evaluation works
correctly
- Added German translation for new error message
- Replaced real user data with privacy-safe dummy data in test fixtures

### 🔧 Type Safety Improvements

**Added explicit `str()` conversions to resolve type inference issues:**

The comprehensive BelenConf test fixture contains deeply nested data
structures that caused pyright's type checker to infer complex
dictionary types throughout the codebase. To ensure type safety and
prevent runtime errors, I added explicit `str()` conversions in key
locations:

- **CSRF tokens**: `str(csrf_token)` - Ensures CSRF tokens are treated
as strings
- **Special attributes**: `str(special_attribute_value)` - Converts
special attribute values to strings
- **DOM attributes**: `str(special_attr_elem.attrs.id)` - Ensures
element IDs are strings
- **URL handling**: `str(current_img_url)` and `str(href_attributes)` -
Converts URLs and href attributes to strings
- **Price values**: `str(ad_cfg.price)` - Ensures price values are
strings

These conversions are defensive programming measures that ensure
backward compatibility and prevent type-related runtime errors, even if
the underlying data structures change in the future.

### ⚙️ Type of Change
- [x]  New feature (adds new functionality without breaking existing
usage)
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)

##  Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
2025-10-12 21:22:46 +02:00
Jens
a2745c03b2 fix: resolve linting errors after dependency updates (#634) 2025-10-12 14:45:59 +02:00
Jens Bergmann
91a40b0116 feat: enhanced folder naming (#599) 2025-08-12 10:43:26 +02:00
Jens Bergmann
1a1633e12d feat: introduce smoke test group and fail-fast test orchestration (#572) 2025-07-09 19:23:52 +02:00
Jens Bergmann
5430f5cdc6 feat: update check (#561)
feat(update-check): add robust update check with interval support, state management, and CLI integration

- Implement version and interval-based update checks with configurable settings
- Add CLI command `kleinanzeigen-bot update-check` for manual checks
- Introduce state file with versioning, UTC timestamps, and migration logic
- Validate and normalize intervals (1d–4w) with fallback for invalid values
- Ensure correct handling of timezones and elapsed checks
- Improve error handling, logging, and internationalization (i18n)
- Add comprehensive test coverage for config, interval logic, migration, and CLI
- Align default config, translations, and schema with new functionality
- Improve help command UX by avoiding config/log loading for `--help`
- Update documentation and README with full feature overview
2025-06-27 07:52:40 +02:00
Heavenfighter
f69ebef643 feat: add new update command to update published ads (#549)
Co-authored-by: Jens Bergmann <1742418+1cu@users.noreply.github.com>
2025-06-16 11:46:51 +02:00
sebthom
23910ffbf5 ci: publish code coverage reports 2025-05-15 22:13:38 +02:00
sebthom
83c0d6adf0 refact: move temp files to /.temp/ 2025-05-15 19:52:41 +02:00
sebthom
85a5cf5224 feat: improve content_hash calculation 2025-05-15 12:07:49 +02:00
sebthom
f1cd597dd8 fix: reduce distribution size 2025-05-15 12:07:49 +02:00
sebthom
6ede14596d feat: add type safe Ad model 2025-05-15 12:07:49 +02:00
sebthom
1369da1c34 feat: add type safe Config model 2025-05-15 12:07:49 +02:00
sebthom
f2e6f0b20b chore: update pyproject.toml 2025-05-12 14:08:50 +02:00
sebthom
314285583e ci: add pip-audit check 2025-05-11 20:14:38 +02:00
sebthom
3e8072973a build: use yamlfix for yaml formatting 2025-04-28 13:17:23 +02:00
sebthom
bda0acf943 refact: enable ruff preview rules 2025-04-28 13:17:23 +02:00
sebthom
ef923a8337 refact: apply consistent formatting 2025-04-28 12:55:28 +02:00
sebthom
fe33a0e461 refact: replace pyright with basedpyright 2025-04-28 12:52:18 +02:00
sebthom
376ec76226 refact: use ruff instead of autopep8,bandit,pylint for linting 2025-04-28 12:51:51 +02:00
sebthom
f0b84ab335 build: simplify pytest config 2025-04-28 12:43:53 +02:00
sebthom
27c7bb56ca fix: downgrading nodriver to 0.39 to address failing browser launch #470 2025-04-07 22:40:41 +02:00
Jens Bergmann
1b004a2a3e Revert "feat: Introduce isort and Python-based code quality tools (#446)"
This reverts commit cfe2b900c7.

The custom scripts introduced to auto-format imports (to enforce project guidelines) caused issues. Specifically, isort’s hardcoded behavior for expanded standard library imports with “as” imports led to unintended formatting. This commit reverts those changes and removes the custom scripts, restoring the project to its previous, stable state.
2025-03-13 11:55:31 +01:00
Jens Bergmann
cfe2b900c7 feat: Introduce isort and Python-based code quality tools (#446) 2025-03-10 06:09:49 +01:00
Jens Bergmann
c7f7b832b2 fix: Make description field optional in ad_defaults
The description field in the main configuration (ad_defaults) is now optional.
Previously, the bot would fail if no description or affixes were provided in
the main configuration. This change addresses issue #435.

Changes:
- Add fallback to empty string ("") when all description prefix/suffix sources
  are None in __get_description_with_affixes method
- Add comprehensive test suite for description handling in test_init.py
- Fix coverage path in pyproject.toml from 'kleinanzeigen_bot' to
  'src/kleinanzeigen_bot'

New tests cover:
- Description handling without main config description
- New format affixes in configuration
- Mixed old/new format affixes
- Ad-level affix precedence
- None value handling in affixes
- Email address @ symbol replacement

This change maintains backward compatibility while making the description
field optional in the main configuration, improving flexibility for users.
2025-02-18 21:39:53 +01:00
sebthom
5ade82b54d chore: update pyproject config 2025-02-10 21:16:38 +01:00
Jens Bergmann
affde0debf test: Enhance test coverage for KleinanzeigenBot initialization and core functionality (#408) 2025-02-09 03:33:01 +01:00
1cu
f4f00b9563 test: Add comprehensive test suite for extract.py (#400) 2025-02-05 23:35:45 +01:00
1cu
76b0901166 test: reorganized unit/integration tests (#398) 2025-02-03 17:05:14 +01:00
sebthom
236740fc2b chore: update pyproject.toml 2025-01-24 22:13:59 +01:00
sebthom
679d08502c chore: regenerate pdm.lock 2024-12-28 19:25:26 +01:00
sebthom
aec051826a chore: update project meta 2024-12-28 18:03:09 +01:00
sebthom
e077f8d86d feat: improve colorized logging 2024-12-27 15:35:58 +01:00
sebthom
9d54a949e7 feat: add multi-language support 2024-12-27 13:04:30 +01:00
sebthom
01d78bb000 feat: support shipping for WANTED ads #349 2024-11-21 23:53:26 +01:00
sebthom
6e76b0ff4c build: rename "scan" script to "audit" 2024-11-21 22:04:15 +01:00
sebthom
1b326c1ce8 chore: upgrade to Python 3.13 and update deps 2024-11-15 13:31:29 +01:00
sebthom
6c5c1940e1 chore: Update Python dependencies 2024-09-16 11:56:34 +02:00
sebthom
0491636666 fix: SSL: CERTIFICATE_VERIFY_FAILED when running compiled version 2024-08-05 13:50:43 +02:00
sebthom
822d3b7e7c upgrade dependencies
- setuptools 69.1.1 -> 69.5.1
- pytest-rerunfailures 13.0 -> 14.0
- autopep8 2.0.4 -> 2.1.0
- typing-extensions 4.10.0 -> 4.11.0
- pyright 1.1.353 -> 1.1.359
- pyinstaller 6.5.0 -> 6.6.0
- pyinstaller-hooks-contrib 2024.3 -> 2024.4
- nodriver 0.27rc3 -> 0.27rc4
2024-04-17 17:49:11 +02:00
sebthom
d5ae070bb3 chore: Update Python dependencies 2024-03-11 00:41:20 +01:00
sebthom
c9f12bfeea add "pdm debug" task 2024-03-07 23:21:50 +01:00
sebthom
a441c5de73 replace selenium with nodriver 2024-03-07 20:33:23 +01:00
sebthom
df92206f8c upgrade pytest 2024-03-04 14:25:35 +01:00
sebthom
115fb82064 remove webdriver-manager dependency 2024-03-04 13:27:17 +01:00
sebthom
9caa7a7124 use venv 2024-03-04 10:07:47 +01:00