mirror/maigret
1
0
Fork 0
mirror of https://github.com/soxoj/maigret.git synced 2026-05-06 22:19:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,262 Commits 15 Branches 34 Tags
0131f0b64cf2e37d542d3324990313e371c2ef9f
Commit Graph

49 Commits

Author SHA1 Message Date
Soxoj 98f03c153b Add 3 crypto sites (Polymarket, Zora, Revolut.me), added crypto inves… (#2538) 
* Add 3 crypto sites (Polymarket, Zora, Revolut.me), added crypto investigation use case page in docs

* Added fintech tag

* Updated sites metadata
 2026-04-21 11:08:48 +02:00
Soxoj fb71f26fd0 Fix site checks: recover 6 CF sites via tls_fingerprint, 500px GraphQL, delete 4 dead domains (#2535) 2026-04-20 22:41:51 +02:00
Soxoj 37ce4fe728 Update of Readme and documentation (#2514) 
* Big readme update

* Readme and documentation update

* Readme structure update

* Small fixes

* Changelog update
 2026-04-17 17:42:36 +02:00
Soxoj 6db1df2ddb Fix failing test for custom DB path resolution (#2468) 
* Fix `--db` bug

* Fix test_resolve_db_path_custom_file to create the file before testing

Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/3ea7b2e8-0565-4fca-8ec2-eff8eb4ee617

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 2026-04-08 00:53:57 +02:00
Copilot 23adc178ea Fix crash on -a --self-check by adding exception handling to site check coroutines (#2466) 
* Initial plan

* Fix crash on -a --self-check by adding exception handling in site_self_check and self_check

Wrap the body of site_self_check in try/except to catch unexpected errors
and always return a valid changes dict. Also add a safety-net try/except
in self_check around awaiting individual site check futures so that a
single site failure doesn't crash the entire self-check process.

Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/5e27d620-5cbb-43d2-a9f9-ecb53a29904d

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>

* Restore @pytest.mark.slow on test_maigret_results

Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/5e27d620-5cbb-43d2-a9f9-ecb53a29904d

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>

* Document --self-check error resilience, --auto-disable, and --diagnose in docs/

Update command-line-options.rst with expanded --self-check description
and new --auto-disable and --diagnose entries. Add a "Database self-check"
section to features.rst explaining error-resilient behaviour and usage
examples. Update usage-examples.rst to reference --auto-disable.

Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/af1f0f09-9112-4902-8475-e81d235ff3ed

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
 2026-04-07 19:44:09 +02:00
Copilot 6ed8fdefcc Add installation troubleshooting for missing system dependencies (#2465) 
* Initial plan

* Add installation troubleshooting section for system dependency errors

Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/6c3a5612-bdd5-4611-ba77-aea7ab52e304

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>

* Simplify README troubleshooting to a link to the full docs

Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/6c557093-0643-4980-93ad-973e2d3141ef

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
 2026-04-07 17:54:02 +02:00
Soxoj ad95302745 Add Markdown reports for LLM analysis (#2463) 2026-04-06 18:26:43 +02:00
Soxoj 269d50eedc DB update mechanism (#2458) 
* Database update mechanism
 2026-04-04 18:00:50 +02:00
Soxoj 5d502eaef6 Add site protection tracking system, fix broken site checks (Instagra… (#2452) 
* Add site protection tracking system, fix broken site checks (Instagram, StackOverflow, LeetCode, Boosty, LiveLib), preserve unicode in data.json

* Update poetry.lock by running poetry lock

Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/14333f41-67d5-4e28-a782-9730b31fc667

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 2026-04-02 20:28:20 +02:00
Soxoj a5d337b765 Tags and site names improvements (#2427) 
- Added social tag to social networks (33 sites)
- Fixed wrong tags (8 sites)
- Filled empty tags for 213 sites in top-1000
- Country tag cleanup (~374 sites)
- Site naming normalization (75 sites)
- New tests (3)
- Documentation updates
 2026-03-28 15:42:12 +01:00
Soxoj b145e7b26f feat(core): add POST request support, new sites, migrate to Majestic Million ranking (#2317) 
* feat(core): add POST request support, new sites, migrate to Majestic Million ranking
- Added native POST request support to the Maigret engine (requestMethod, requestPayload) to enable querying modern JSON registration endpoints.
- Replaced the discontinued Alexa rank API with the Majestic Million dataset for global popularity sorting and automated CI updates.
- Fixed multiple false positives among top 500 sites and bypassed standard anti-bot protections using custom User-Agents.
- Updated public documentation and internal playbooks to reflect the new features.

* feat(data): apply all data.json site check updates from main branch

- Added CTFtime and PentesterLab (new sites added in main)
- Removed forums.imore.com (deleted in main as dead site)
- Disabled 5 sites per main branch fixes: Librusec, MirTesen, amateurvoyeurforum.com, forums.stevehoffman.tv, vegalab
- Fixed 5 site checks per main branch: SoundCloud, Taplink, Setlist, RoyalCams, club.cnews.ru (switched from status_code to message checkType with proper markers)

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/a1d194d9-c0ff-4e2b-974c-c5e4b59548bf

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 2026-03-24 22:08:42 +01:00
Copilot 2e430e5039 feat: add tag blacklisting via --exclude-tags (#2352) 
* Initial plan

* feat: add tag blacklisting support (--exclude-tags CLI flag, web UI, docs, tests)

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/1a656af2-36bf-494f-9f03-1b5340f0357c

* fix: correct tag cloud label to match click-cycle interaction

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/1a656af2-36bf-494f-9f03-1b5340f0357c

* feat: add all country tags to web interface tag cloud

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/7e184b24-ff26-48fd-8a93-aea12b0a8d7b

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
 2026-03-24 22:00:59 +01:00
Soxoj 97cc4b46d9 Improve site-check quality: fix broken site configs, add diagnostic utilities, and make self-check report-only by default with opt-in auto-disable. (#2301) 
- Fix VK and TradingView checkType; add Reddit and Microsoft Learn API-style probes where appropriate; adjust or disable entries that are unreliable under anti-bot protection.
- Self-check: stop aggressive auto-disable; default to reporting issues only; add --auto-disable and --diagnose for optional fixes and deeper output.
- Tooling: add utils/site_check.py and utils/check_top_n.py (and related helpers) to inspect and rank site behavior against the top-N list
- Scope: aligns with fixing top-traffic / high-impact sites and making diagnostics repeatable without silently flipping disabled flags
 2026-03-22 16:48:35 +01:00
Soxoj 227a25bfa1 Twitter fixed, mirrors mechanism improvement (#2299) 2026-03-22 01:14:17 +01:00
Soxoj f99091f5f7 Fixed false positives in top-500 (#2292) 2026-03-21 23:35:59 +01:00
Soxoj bebadb0362 Bump to 0.5.0 (#2108) 2025-08-10 13:10:50 +02:00
Ikko Eltociear Ashimine 4195a3ca21 docs: update usage-examples.rst (#1996) 
inital -> initial
 2025-02-18 10:50:29 +01:00
imgbot[bot] 900ed840b3 [ImgBot] Optimize images (#1974) 
*Total -- 2,762.12kb -> 2,382.30kb (13.75%)

/static/web_interface_screenshot_start.png -- 106.23kb -> 59.10kb (44.37%)
/docs/source/maigret_screenshot.png -- 375.21kb -> 233.98kb (37.64%)
/static/web_interface_screenshot.png -- 615.65kb -> 424.23kb (31.09%)
/static/recursive_search.svg -- 1,665.02kb -> 1,664.99kb (0%)

Signed-off-by: ImgBotApp <ImgBotHelp@gmail.com>
Co-authored-by: ImgBotApp <ImgBotHelp@gmail.com>
 2024-12-16 19:24:08 +01:00
Soxoj c3dfe9cb4d Small docs and parameters fixes for web interface mode (#1973) 2024-12-16 17:18:22 +01:00
Soxoj 4894a267d7 Added web interface docs (#1972) 2024-12-16 17:06:06 +01:00
Soxoj cb01535565 Preparation of 0.5.0 alpha version (#1966) 2024-12-13 12:51:31 +01:00
Soxoj 4b1317789d Refactored self-check method, code formatting, small lint fixes (#1942) 2024-12-07 18:05:30 +01:00
Soxoj f4edab8946 Readme/docs update based on GH discussions (#1936) 2024-12-06 01:54:26 +01:00
Soxoj f04de78682 Activation mechanism documentation added (#1935) 
Few site checks fixed
 2024-12-06 01:35:19 +01:00
Soxoj e701c881a1 Documentation update (#1931) 2024-12-04 16:58:56 +01:00
Soxoj e982be4109 Installation docs update (#1927) 2024-12-03 20:23:49 +01:00
Soxoj 07a7a474f8 Documentation update (#1926) 2024-12-03 17:25:17 +01:00
Soxoj 2f93963a0a Refactored sites module, updated documentation (#1918) 2024-12-01 11:41:41 +01:00
Soxoj e96d09dee7 Permutator output and documentation updates (#1914) 2024-11-29 13:15:03 +01:00
Soxoj 86d51bced0 Added 7 sites, implemented integration with Marple, docs update (#1881) 
* Added 5 sites, implemented integration with Marple

* Added 2 more sites, updated docs

* Updated sites list
 2024-11-25 14:41:34 +01:00
Soxoj 24e545b62c Added dev documentation, fixed some sites, removed GitHub issue links from reports (#1869) 2024-11-23 18:45:56 +01:00
Nadeem M 70bed56a8a Update philosophy.rst (#866) 2023-04-12 16:51:48 +02:00
Soxoj 79afab11c2 Fixed docs about tags (#715) 2022-10-17 00:44:00 +03:00
Lorenzo Sapora 95276b841c Fix typos (#681) 2022-10-04 09:16:41 +03:00
Leon G 06f94cd476 correct username in usage examples (#673) 2022-10-04 00:40:49 +03:00
Soxoj f7c7809d8d Bump to 0.4.4 (#621) 2022-09-03 14:30:24 +03:00
Soxoj 34073d12f4 XMind 8 report warning and some docs update (#452) 2022-04-23 01:28:31 +03:00
Soxoj 19956f74ca Bump to 0.4.3 2022-04-13 22:58:21 +03:00
cyb3rk0tik 8c007219f5 Bump to 0.4.2 2022-03-07 21:42:34 +05:00
Soxoj 8a865a1ce6 Op.gg fixes (#363) 
* Fixed op.gg sites

* Added testing docs, fixed some error

* Updated site list and statistics
 2022-02-26 14:16:13 +03:00
Soxoj 31fc656721 Added package publishing instruction (#356) 2022-02-23 16:46:58 +03:00
Soxoj 58ae979904 Bumped to v0.4.1 (#317) 2022-01-15 12:51:27 +03:00
Soxoj e520418f6a Bump to v0.4.0 (#287) 2022-01-03 01:44:11 +03:00
Soxoj 1b9ce3bac7 Removed redundant logging 2021-11-07 18:52:56 +03:00
soxoj 4ff19970dd Merge branch 'main' into settings-new-configs 2021-11-06 14:31:29 +03:00
Soxoj 267d9e505b New config options added 2021-11-06 14:29:56 +03:00
Soxoj 7a362406d5 Documentation and settings improved 2021-11-06 08:24:12 +03:00
soxoj 4ce241893b Created ReadTheDocs requirements file 2021-11-02 20:22:00 +03:00
Egor Nagornov eeab6ba82c Move wiki pages to ReadTheDocs 2021-11-02 17:21:27 +07:00