Commit Graph

63 Commits

Author SHA1 Message Date
Soxoj 97cc4b46d9 Improve site-check quality: fix broken site configs, add diagnostic utilities, and make self-check report-only by default with opt-in auto-disable. (#2301)
- Fix VK and TradingView checkType; add Reddit and Microsoft Learn API-style probes where appropriate; adjust or disable entries that are unreliable under anti-bot protection.
- Self-check: stop aggressive auto-disable; default to reporting issues only; add --auto-disable and --diagnose for optional fixes and deeper output.
- Tooling: add utils/site_check.py and utils/check_top_n.py (and related helpers) to inspect and rank site behavior against the top-N list
- Scope: aligns with fixing top-traffic / high-impact sites and making diagnostics repeatable without silently flipping disabled flags
2026-03-22 16:48:35 +01:00
Soxoj 227a25bfa1 Twitter fixed, mirrors mechanism improvement (#2299) 2026-03-22 01:14:17 +01:00
Soxoj f99091f5f7 Fixed false positives in top-500 (#2292) 2026-03-21 23:35:59 +01:00
Soxoj 97e5f600d0 Async generator-executor for site checks (#1978) 2024-12-17 22:48:11 +01:00
Soxoj 4eada16b94 Added a test for submitter (#1944) 2024-12-08 13:35:27 +01:00
Soxoj c66d776f8a Refactoring, test coverage increased to 60% (#1943) 2024-12-08 02:13:28 +01:00
Soxoj 4b1317789d Refactored self-check method, code formatting, small lint fixes (#1942) 2024-12-07 18:05:30 +01:00
Soxoj cd789ed138 Fixed Ebay and BongaCams checks (#1939) 2024-12-06 13:32:51 +01:00
Soxoj 5641456ba0 Weibo site check fix, activation mechanism added (#1938) 2024-12-06 11:31:20 +01:00
Soxoj cb9f01c106 Fixed Figma check (#1932)
Fixed cookies bug
Improved self-check mode: don't disable sites because of check errors
2024-12-04 19:21:27 +01:00
Soxoj 2f93963a0a Refactored sites module, updated documentation (#1918) 2024-12-01 11:41:41 +01:00
Soxoj 2e2a47a12b Close http connections (#1595) (#1905) 2024-11-27 15:28:10 +01:00
Soxoj ee25c61fc2 Maigret bot support (custom progress function fixed) (#1898)
* Fixed progress/close functions
* Fixed tests: execution time increased with alive_progressbar
2024-11-26 15:54:26 +01:00
Soxoj 324c118530 Parallel execution optimization (#1897)
* Connection failure fix: removed futures, added semaphores

* Additional fixes

* Tqdm replace to alive_progress, poetry update

* Self-check mode fix, tests fixes

* Sites checks fixes (#1896)

* Fixed incorrect site names, added method to compare sites
2024-11-26 13:55:12 +01:00
Soxoj 13c20afe5b Improved self-check mode (#1887) 2024-11-25 18:27:59 +01:00
weekend sorrow f1ea12d731 Updating site checkers, disabling suspended sites (#1266)
* Fixing checks for broken sites and repairing the ones that were changed

* little tweaks

* little tweaks

---------

Co-authored-by: Weekrow <somewherelse@yandex.ru>
2023-10-27 21:43:45 +02:00
Soxoj f5ca005766 Added memory.lol (Twitter usernames archive) (#1067) 2023-07-24 12:57:45 +06:00
Soxoj 0ad2cdef2c Fixed false positives, updated networkx dep, some lint fixes (#894)
* Fixed false positives, updated networkx dep, some lint fixes

* Downgraded networkx version
2023-04-16 18:24:29 +02:00
Soxoj 356d7d4e49 Fixed documentation URL (#799) 2022-12-18 12:26:19 +03:00
fen0s 1fd1e2c809 Mirrors (#630)
* Update checking.py

* Added attempts check and mirrors

Co-authored-by: Soxoj <soxoj@protonmail.com>
Co-authored-by: Soxoj <31013580+soxoj@users.noreply.github.com>
2022-09-11 14:05:32 +03:00
Soxoj 929366cc81 Improved usability of external progressbar func (#476) 2022-05-14 02:06:33 +03:00
Soxoj 8a865a1ce6 Op.gg fixes (#363)
* Fixed op.gg sites

* Added testing docs, fixed some error

* Updated site list and statistics
2022-02-26 14:16:13 +03:00
Soxoj 2c33d797ce Temporary fix for Twitter (#299) (#300) 2022-01-08 19:15:57 +03:00
Soxoj 262f97ce33 Fixed xHamster, added support of proxies to self-check mode (#259) 2021-12-10 23:01:56 +03:00
soxoj 7decbce08d Box drawing logic improved, added new settings (#217) 2021-11-09 02:33:31 +03:00
Soxoj e3b4512c47 Delayed import of requests module, speed check command, reqs updated 2021-10-31 19:37:26 +03:00
Soxoj 6480eebbdf Accelerated start time & fixed some false positives 2021-10-31 18:25:01 +03:00
Soxoj c7977f1cdf Draft of graph report 2021-06-21 22:20:51 +03:00
Soxoj eb721dc7e3 Makefile, some fixes 2021-06-06 17:32:04 +03:00
Soxoj 02cf330e37 Added I2P sites support 2021-06-02 23:45:11 +03:00
Soxoj 13e1b6f4d1 Added some domains for new DNS checker, fixed reports generation crashes 2021-06-02 23:16:44 +03:00
Soxoj 1a2c7e944a Added several sites 2021-06-01 00:28:14 +03:00
Soxoj ba6169659e Added DNS checker 2021-05-22 03:12:04 +03:00
Soxoj a76f95858f Added checker of Tor sites 2021-05-20 23:26:02 +03:00
Soxoj 2c04ccce57 Improved tests 2021-05-18 00:43:56 +03:00
Soxoj b696b982f4 Added some photo sites, improved errors detecting 2021-05-08 20:37:34 +03:00
Soxoj 53e9dab677 Added Yelp and yelp_userid support 2021-05-08 03:34:03 +03:00
Soxoj 960708ef2e Facebook parsing fixed, website field added 2021-05-08 02:25:54 +03:00
Soxoj 2145027196 Added several sites, some improvements 2021-05-07 01:20:20 +03:00
Soxoj 3cbb9df7b3 Main maigret function refactoring 2021-05-05 18:02:13 +03:00
Soxoj 3b91a9cd31 CLI arguments improvements, tests added 2021-05-05 15:27:56 +03:00
Soxoj c88e194d07 Refactored to decrease cyclomatic complexity 2021-05-05 10:55:33 +03:00
Soxoj 66d6c7a93c Added some new sites, engines updates 2021-05-03 03:16:02 +03:00
Soxoj 5ee91f6659 Introduced --retries flag, made thorough refactoring
- updated sites list
- test scripts linting
2021-05-01 23:54:01 +03:00
Soxoj bfa6afac32 Refactoring and linting, added notifications about frequent search errors 2021-04-30 12:03:13 +03:00
Soxoj c9194b20ba Added some new sites, implemented filtering by source site with --name, improved submit mode 2021-04-29 17:11:17 +03:00
Soxoj 2cdc9bb276 Added some new sites and introduced 'source' feature 2021-04-29 15:14:21 +03:00
Soxoj f43dc5bd6f Errors stats MVP, some fp fixes 2021-04-25 01:08:23 +03:00
Soxoj 2fee65fe4e Added asyncio tasks with timeouts, non-blocking work with queue 2021-04-11 17:56:27 +03:00
Soxoj b755628a1d Documentation and API improving 2021-03-30 00:19:17 +03:00