Commit Graph

69 Commits

Author SHA1 Message Date
Jens
9ed87ff17f fix: wait for image upload completion before submitting ad (#716)
## ℹ️ Description
Fixes a race condition where ads were submitted before all images
finished uploading to the server, causing some images to be missing from
published ads.

- Link to the related issue(s): Issue #715
- The bot was submitting ads immediately after the last image
`send_file()` call completed, only waiting 1-2.5 seconds via
`web_sleep()`. This wasn't enough time for server-side image processing,
thumbnail generation, and DOM updates to complete, resulting in missing
images in published ads.

## 📋 Changes Summary

### Image Upload Verification (Initial Fix)
- Added thumbnail verification in `__upload_images()` method to wait for
all image thumbnails to appear in the DOM after upload
- Added configurable timeout `image_upload` to `TimeoutConfig` (default:
30s, minimum: 5s)
- Improved error messages to show expected vs actual image count when
upload times out
- Added German translations for new log messages and error messages
- Regenerated JSON schemas to include new timeout configuration

### Polling Performance & Crash Fix (Follow-up Fix)
- Fixed critical bug where `web_find_all()` would raise `TimeoutError`
when no thumbnails exist yet, causing immediate crash
- Wrapped DOM queries in `try/except TimeoutError` blocks to handle
empty results gracefully
- Changed polling to use `self._timeout("quick_dom")` (~1s with PR #718)
instead of default timeout
- Improved polling performance: reduced cycle time from ~2s to ~1.5s
- DOM queries are client-side only (no server load from frequent
polling)

**New configuration option:**
```yaml
timeouts:
  image_upload: 30.0  # Total timeout for image upload and server-side processing
  quick_dom: 1.0      # Per-poll timeout for thumbnail checks (adjustable via multiplier)
```

The bot now polls the DOM for `ul#j-pictureupload-thumbnails >
li.ui-sortable-handle` elements after uploading images, ensuring
server-side processing is complete before submitting the ad form.

### ⚙️ Type of Change
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ]  New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)


##  Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Image uploads now verify completion by waiting for all uploaded
thumbnails to appear before proceeding.

* **Improvements**
  * Added a configurable image upload timeout (default 30s, minimum 5s).
* Improved timeout reporting: when thumbnails don’t appear in time, the
app returns clearer feedback showing expected vs. observed counts.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-10 14:04:42 +01:00
Jens
6cbc25b54c docs: Improve README clarity and fix configuration documentation (#711)
## ℹ️ Description

This PR addresses issue #708 by improving the README's About section to
make the bot's purpose clearer to new users. It also fixes a technical
inaccuracy in the configuration documentation.

- Link to the related issue(s): Issue #708
- **Motivation**: The current About section uses ambiguous terminology
("ads" instead of "listings") and doesn't clearly communicate what the
bot does. Additionally, the configuration example incorrectly documents
`shipping_costs` as available in `ad_defaults`, when it's only
implemented for per-ad configuration.

## 📋 Changes Summary

**About Section Improvements:**
- Changed "ads" to "listings" for clarity (addresses confusion mentioned
in #708)
- Added "Key Features" section with 6 concrete capabilities
- Added "Why This Project?" section explaining the rewrite and
differences from legacy client
- Preserved all legal disclaimers

**Configuration Documentation Fix:**
- Removed `shipping_costs` from `ad_defaults` example (not implemented
in `AdDefaults` Pydantic class)
- Added clarifying comment that `shipping_costs` and `shipping_options`
must be configured per-ad
- Verified `shipping_costs` remains documented in ad configuration
section

### ⚙️ Type of Change
- [x] 🐞 Bug fix (non-breaking change which fixes an issue)
- [ ]  New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)

*Note: This is a documentation-only change with no code modifications.*

##  Checklist
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (N/A -
documentation only).
- [x] I have formatted the code (N/A - documentation only).
- [x] I have verified that linting passes (N/A - documentation only).
- [x] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
2025-12-05 20:51:48 +01:00
Jens
a3ac27c441 feat: add configurable timeouts (#673)
## ℹ️ Description
- Related issues: #671, #658
- Introduces configurable timeout controls plus retry/backoff handling
for flaky DOM operations.

We often see timeouts which are note reproducible in certain
configurations. I suspect timeout issues based on a combination of
internet speed, browser, os, age of the computer and the weather.

This PR introduces a comprehensive config model to tweak timeouts.

## 📋 Changes Summary
- add TimeoutConfig to the main config/schema and expose timeouts in
README/docs
- wire WebScrapingMixin, extractor, update checker, and browser
diagnostics to honor the configurable timeouts and retries
- update translations/tests to cover the new behaviour and ensure
lint/mypy/pyright pipelines remain green

### ⚙️ Type of Change
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [x]  New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)

##  Checklist
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Centralized, configurable timeout system for web interactions,
detection flows, publishing, and pagination.
* Optional retry with exponential backoff for operations that time out.

* **Improvements**
* Replaced fixed wait times with dynamic timeouts throughout workflows.
  * More informative timeout-related messages and diagnostics.

* **Tests**
* New and expanded test coverage for timeout behavior, pagination,
diagnostics, and retry logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-13 15:08:52 +01:00
Jens Bergmann
332926519d feat: chrome version detection clean (#607) 2025-08-18 13:19:50 +02:00
Jens Bergmann
c9d04da70d feat: browser connection improvements (#601) 2025-08-13 09:29:25 +02:00
Jens Bergmann
91a40b0116 feat: enhanced folder naming (#599) 2025-08-12 10:43:26 +02:00
Jens Bergmann
c425193b10 feat: add create-config subcommand to generate default config (#577) 2025-07-13 13:09:40 +02:00
Jens Bergmann
1a1633e12d feat: introduce smoke test group and fail-fast test orchestration (#572) 2025-07-09 19:23:52 +02:00
Jens Bergmann
7ff005d18b fix: chores (#565) 2025-07-03 15:12:43 +02:00
Jens Bergmann
5430f5cdc6 feat: update check (#561)
feat(update-check): add robust update check with interval support, state management, and CLI integration

- Implement version and interval-based update checks with configurable settings
- Add CLI command `kleinanzeigen-bot update-check` for manual checks
- Introduce state file with versioning, UTC timestamps, and migration logic
- Validate and normalize intervals (1d–4w) with fallback for invalid values
- Ensure correct handling of timezones and elapsed checks
- Improve error handling, logging, and internationalization (i18n)
- Add comprehensive test coverage for config, interval logic, migration, and CLI
- Align default config, translations, and schema with new functionality
- Improve help command UX by avoiding config/log loading for `--help`
- Update documentation and README with full feature overview
2025-06-27 07:52:40 +02:00
Heavenfighter
f69ebef643 feat: add new update command to update published ads (#549)
Co-authored-by: Jens Bergmann <1742418+1cu@users.noreply.github.com>
2025-06-16 11:46:51 +02:00
sebthom
23910ffbf5 ci: publish code coverage reports 2025-05-15 22:13:38 +02:00
sebthom
6ede14596d feat: add type safe Ad model 2025-05-15 12:07:49 +02:00
sebthom
1369da1c34 feat: add type safe Config model 2025-05-15 12:07:49 +02:00
sebthom
f2e6f0b20b chore: update pyproject.toml 2025-05-12 14:08:50 +02:00
Airwave1981
d87ae6e740 feat: allow auto-restart on captcha (#481)
Co-authored-by: sebthom <sebthom@users.noreply.github.com>
2025-04-26 14:40:47 +02:00
Jens Bergmann
772326003f fix: Separate 'changed' and 'due' ad selectors (#442)
This commit implements a new 'changed' selector for the --ads option that
publishes only ads that have been modified since their last publication.
The 'due' selector now only republishes ads based on the time interval,
without considering content changes.

The implementation allows combining selectors with commas (e.g., --ads=changed,due)
to publish both changed and due ads. Documentation and translations have been
updated accordingly.

Fixes #411
2025-02-28 14:53:53 -05:00
sebthom
9f19cd85bd docs: fix build status badge 2025-02-12 21:40:45 +01:00
Jens Bergmann
4051620aed enh: allow per-ad overriding of global description affixes (#416) 2025-02-11 23:39:26 +01:00
sebthom
e8d342dc68 docs: document ad config defaults 2025-02-10 03:23:33 +01:00
sebthom
08197eabae docs: improve disclaimer 2025-02-03 22:06:30 +01:00
1cu
f01109c956 feat: add hash-based ad change detection (#343) (#388)
Co-authored-by: sebthom <sebthom@users.noreply.github.com>
2025-01-26 23:37:33 +01:00
sebthom
3d27755207 docs: update README 2025-01-26 20:05:28 +01:00
sebthom
7d9b857a46 docs: Update doc 2025-01-20 21:40:28 +01:00
Heavenfighter
ca876e628b fix shipping options when downloading. Fixes #375 (#376) 2025-01-10 16:05:11 +01:00
sebthom
1e782beabc fix: update help text 2024-12-27 13:49:05 +01:00
sebthom
9d54a949e7 feat: add multi-language support 2024-12-27 13:04:30 +01:00
sebthom
a738f0748d docs: add related OSS projects section 2024-12-22 22:19:44 +01:00
sebthom
a876add5a7 feat: by default delete old ads after republishing #338 2024-11-22 12:41:34 +01:00
sebthom
f9fdf4d158 refact: update categories 2024-11-22 12:40:27 +01:00
Moritz Graf
b3cc8ef5cd Fixing missing src directory in README (#288) 2024-04-01 18:09:50 +02:00
sebthom
e7c7ba90be support re-using already open browser window 2024-03-07 23:07:23 +01:00
sebthom
a441c5de73 replace selenium with nodriver 2024-03-07 20:33:23 +01:00
sebthom
3a52f7f629 update doc 2024-01-12 11:37:40 +01:00
sebthom
b723e59ac4 update kleinanzeigen URL 2023-05-15 21:45:57 +02:00
Jeppy
70868395b2 FIX typo 2023-02-25 12:04:34 +01:00
Philipp K
886d812393 FIX Download path not found 2022-12-18 20:11:57 +01:00
Jeppy
45fd4bac9f ADD download shipping options and sell directly
LIMITATION extracts only the cheapest shipping option
2022-12-12 13:18:00 +01:00
Jeppy
e483c003d0 ADD ability to select shipping packages and sell directly 2022-12-12 13:18:00 +01:00
PhilK-7
70a23dbcc7 ADD download --all feature (#139) 2022-11-22 18:52:32 +01:00
PhilK-7
5321b7cd12 Add download command. Fixes #32 (#114) 2022-10-25 22:27:10 +02:00
Peter Simon
728d2cd888 Executable bit for user needs 755 2022-10-03 10:32:32 +02:00
sebthom
1e3c0bee24 add code of conduct 2022-08-21 20:29:55 +02:00
sebthom
f924ed89d0 Support extended glob patterns 2022-06-11 18:55:13 +02:00
sebthom
3964a3a1d3 Allow setting price-type to NOT_APPLICABLE. Addresses #52 2022-03-31 23:26:57 +02:00
sebthom
05cfcdeec8 update doc 2022-03-17 14:52:32 +01:00
sebthom
2597b902be add --keep-old flag 2022-03-12 18:55:49 +01:00
sebthom
15ffb5d402 Update badge 2022-03-12 15:05:21 +01:00
sebthom
7c08214f10 add support for shipping_costs 2022-03-12 13:26:39 +01:00
Oleg Loewen
686b05027d Extend/fix special attributes support (#35) 2022-03-09 16:49:22 +01:00