mirror/maigret
1
0
Fork 0
mirror of https://github.com/soxoj/maigret.git synced 2026-05-07 06:24:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,162 Commits 16 Branches 34 Tags
9b35fc1ab04e24fafa9f4f3564acaf05d549fef1
Commit Graph

435 Commits

Author SHA1 Message Date
Copilot 9b35fc1ab0 [WIP] Fix false-positive probe for vegalab site (#2336) 
* Initial plan

* Disable vegalab site check: domain is dead (DNS does not resolve), causing false positives

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/98430e81-5dcb-4cb3-9aaa-f8c5ce86d026

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
 2026-03-23 20:09:46 +01:00
Copilot 146bc0481b Disable forums.stevehoffman.tv due to false positives (#2331) 
* Initial plan

* Disable forums.stevehoffman.tv to fix false-positive site probe

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/39fea4a9-ec6d-4a12-b34b-1a3486d647e4

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
 2026-03-23 20:08:15 +01:00
Copilot 5930a3022e Disable false-positive site probe: amateurvoyeurforum.com (#2332) 
* Initial plan

* Disable amateurvoyeurforum.com site check to fix false positives

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/e7fcad2b-4511-4e6d-b186-411951170e0a

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
 2026-03-23 20:07:42 +01:00
Copilot b1a211c3cd Disable forums.developer.nvidia.com (auth-gated user profiles) (#2305) 
* Initial plan

* disable forums.developer.nvidia.com due to auth-locked user pages

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/b8f41f15-8588-4aac-a443-af5e2aaa1918

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
 2026-03-22 22:43:51 +01:00
Copilot 56d0c9f2f1 Remove dead site xxxforum.org (#2310) 
* Initial plan

* Remove broken site xxxforum.org from data.json and sites.md

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/bfbd3aa8-bfb1-480a-b2e7-a2c40fc69def

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
 2026-03-22 22:43:21 +01:00
Copilot 01049b730d Fix Love.Mail.ru: update to numeric-only identifiers and new profile URL (#2307) 
* Initial plan

* fix: update Love.Mail.ru to use numeric-only identifiers (#1264)

- Add regexCheck to enforce numeric-only IDs (^\d+$)
- Update usernameClaimed/usernameUnclaimed to numeric values
- Site remains disabled pending live verification

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/6de16097-6bc1-424a-beb1-1d2ec6b99944

* fix: update Love.Mail.ru URL to /profile/ path, enable check with verified ID

Use maintainer-provided working link https://love.mail.ru/profile/1838153357.
- Change URL pattern from /ru/{username} to /profile/{username}
- Set usernameClaimed to 1838153357
- Remove disabled flag to enable the check

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/ac07d38e-46e2-42d3-9e93-eda3e5cfbcc3

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
 2026-03-22 22:42:59 +01:00
Copilot 4f397fed1c Re-enable taplink.cc with browser User-Agent to bypass Cloudflare (#2308) 
* Initial plan

* fix(taplink): re-enable taplink.cc with browser User-Agent header to bypass Cloudflare

Remove disabled flag and add a Chrome User-Agent header to help
bypass Cloudflare bot detection for taplink.cc profile checks.
If Cloudflare still blocks requests, maigret's built-in error
detection will gracefully mark results as UNKNOWN.

Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
Agent-Logs-Url: https://github.com/soxoj/maigret/sessions/271904b6-e358-4aeb-b503-21c9b91186d9

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: soxoj <31013580+soxoj@users.noreply.github.com>
 2026-03-22 22:10:44 +01:00
Soxoj 959b2be136 feat(sites): fix false positives: disable 74 broken sites, fix 8 with API probes and better markers (#2302) 
- Disable 74 sites: Cloudflare/captcha blocks, identical responses,
    dead domains, vBulletin/phpBB engine failures
  - Fix Roblox, Salon24.pl, Planetaexcel → status_code (clear 404 signal)
  - Fix en.brickimedia.org → message with "noarticletext" absenceStr
  - Fix Arduino → narrower title-based presenseStrs/absenceStrs
  - Re-enable Fandom (3 wikis) via MediaWiki api.php urlProbe
  - Re-enable Substack via /api/v1/user/{}/public_profile urlProbe
  - Re-enable hashnode via GraphQL GET urlProbe (URL-encoded query)
  - Document lessons: engine template drift, search-by-author fragility,
    always-200 sites, TLS degradation, API bypassing Cloudflare,
    GraphQL GET support, URL-encoding for template safety
 2026-03-22 20:47:51 +01:00
Soxoj 97cc4b46d9 Improve site-check quality: fix broken site configs, add diagnostic utilities, and make self-check report-only by default with opt-in auto-disable. (#2301) 
- Fix VK and TradingView checkType; add Reddit and Microsoft Learn API-style probes where appropriate; adjust or disable entries that are unreliable under anti-bot protection.
- Self-check: stop aggressive auto-disable; default to reporting issues only; add --auto-disable and --diagnose for optional fixes and deeper output.
- Tooling: add utils/site_check.py and utils/check_top_n.py (and related helpers) to inspect and rank site behavior against the top-N list
- Scope: aligns with fixing top-traffic / high-impact sites and making diagnostics repeatable without silently flipping disabled flags
 2026-03-22 16:48:35 +01:00
Soxoj 227a25bfa1 Twitter fixed, mirrors mechanism improvement (#2299) 2026-03-22 01:14:17 +01:00
Soxoj f99091f5f7 Fixed false positives in top-500 (#2292) 2026-03-21 23:35:59 +01:00
Soxoj fb26ccd1f6 Disabled some sites giving false positive results (#2170) 2025-08-22 03:10:47 +02:00
Soxoj bebadb0362 Bump to 0.5.0 (#2108) 2025-08-10 13:10:50 +02:00
MR-VL d90d8a8ac9 Disable AskFM (#2037) 2025-07-13 16:16:49 +02:00
Darlyson Rangel c9e38632ca Disable ICQ site (#1993) 2025-06-28 23:46:09 +02:00
Pierre-Yves Lapersonne f76ea5d738 [#2010] Add 6 more websites to manage (#2009) 
* feat: add `framapiaf.org` in supported web sites, add tag `mastodon` (#2010)

Signed-off-by: Pierre-Yves Lapersonne <dev@pylapersonne.info>

* feat: add `write.as` in supported web sites, add tag `writefreely` (#2010)

Signed-off-by: Pierre-Yves Lapersonne <dev@pylapersonne.info>

* feat: add `programming.dev` in supported web sites, add tag `lemmy` (#2010)

Signed-off-by: Pierre-Yves Lapersonne <dev@pylapersonne.info>

* feat: add `mamot.fr` in supported web sites (#2010)

Signed-off-by: Pierre-Yves Lapersonne <dev@pylapersonne.info>

* feat: add `pixelfed.social` in supported web sites, add tag `pixelfed` (#2010)

Signed-off-by: Pierre-Yves Lapersonne <dev@pylapersonne.info>

* feat: add `Outgress` in supported web sites (#2010)

Signed-off-by: Pierre-Yves Lapersonne <dev@pylapersonne.info>

* Updated the list of supported sites

---------

Signed-off-by: Pierre-Yves Lapersonne <dev@pylapersonne.info>
Co-authored-by: Soxoj <soxoj@protonmail.com>
 2025-06-28 23:33:29 +02:00
Soxoj 97e5f600d0 Async generator-executor for site checks (#1978) 2024-12-17 22:48:11 +01:00
Soxoj f212bc9bc8 Site check fixes 2024-12-12 21:39:35 +01:00
Soxoj 4dd82bf4c9 Fixed Gravatar parsing (socid_extractor) 2024-12-12 02:30:29 +01:00
Soxoj 64ae391a4a Updated Vimeo, CNET, DailyMotion 2024-12-11 01:17:20 +01:00
Soxoj 127d9032c3 Fixed Vimeo, activation/probing mechanisms improvements 2024-12-11 00:56:00 +01:00
Soxoj 81a817a39f Improved "submit new site" mode, added tests, fixed top-500 sites (#1952) 2024-12-10 18:02:43 +01:00
Soxoj 51ab988e36 Fixed ProductHunt check (#1951) 2024-12-09 17:06:03 +01:00
Soxoj 5517636850 Updated OP.GG checks (#1950) 
* Updated OP.GG checks
* Finalized LoL, added Valorant, disabled Archive.org
 2024-12-09 15:59:19 +01:00
Soxoj c66d776f8a Refactoring, test coverage increased to 60% (#1943) 2024-12-08 02:13:28 +01:00
Soxoj 4b1317789d Refactored self-check method, code formatting, small lint fixes (#1942) 2024-12-07 18:05:30 +01:00
Soxoj 8b7d8073d9 Fixed Linktr and discourse.mozilla.org (#1941) 2024-12-07 17:11:39 +01:00
Soxoj 2aa1ea39a0 Site fixes (#1940) 2024-12-06 14:27:38 +01:00
Soxoj cd789ed138 Fixed Ebay and BongaCams checks (#1939) 2024-12-06 13:32:51 +01:00
Soxoj 5641456ba0 Weibo site check fix, activation mechanism added (#1938) 2024-12-06 11:31:20 +01:00
Soxoj f04de78682 Activation mechanism documentation added (#1935) 
Few site checks fixed
 2024-12-06 01:35:19 +01:00
Soxoj cb9f01c106 Fixed Figma check (#1932) 
Fixed cookies bug
Improved self-check mode: don't disable sites because of check errors
 2024-12-04 19:21:27 +01:00
Soxoj 1cb25946dd Disabled Figma check (#1928) 2024-12-04 00:27:55 +01:00
Soxoj d15e12750b Sites fixes (#1917) 
* Some sites fixes

* Sites stats updated
 2024-12-01 03:19:36 +01:00
Soxoj b370bc4c44 Sites checks fixes (#1896) 
Fixed incorrect site names, added method to compare sites
 2024-11-26 13:29:43 +01:00
Soxoj d8a05807ba New sites added (#1888) 2024-11-25 18:24:20 +01:00
Soxoj 86d51bced0 Added 7 sites, implemented integration with Marple, docs update (#1881) 
* Added 5 sites, implemented integration with Marple

* Added 2 more sites, updated docs

* Updated sites list
 2024-11-25 14:41:34 +01:00
Soxoj 54b864f167 Disabled unavailable sites (#1880) 2024-11-24 17:19:31 +01:00
Soxoj 24e545b62c Added dev documentation, fixed some sites, removed GitHub issue links from reports (#1869) 2024-11-23 18:45:56 +01:00
synth 05db32f28f Fixed 1 site, PyInstaller workflow, Google Colab example (#1558) 
* Updated example colab file (Due to latest update)

* Fix RobertsSpaceIndustries URI

* Fix PyInstaller workflow

* Fix example.ipynb (read desc.)

Currently the version installed via pip3 doesn't appear to contain the latest data.json file, resulting in many false positives..

* Fix non-existant users (read desc.)

Fixed non-existant usernames for the following:
Telegram (t.me)
TikBuddy (tikbuddy.com)
FurAffinity (furaffinity.net)
 2024-10-21 15:58:16 +02:00
Paul Pfeister 1cb589eadb chore: remove alik.cz (#1671) 
Alik.cz is seeing unusually high traffic on usernames julian and
noonewouldeverusethis due to its presence in both Sherlock and Maigret.
This target is permanently removed and should not be replaced.
 2024-10-21 15:57:11 +02:00
ranlo e02a5571b6 Update data.json (#1559) 
changed the URL for vidamora.com to www.vidamora.com

any username on https://vidamora.com/profile/{username} returns a redirect, to www.vidamora.com

on https://www.vidamora.com, you get different behavior for existing and non-existing users.
 2024-06-24 12:51:54 +02:00
Richard Mwewa 034153791b Fixed 3 sites, disabed 3, added (#1539) 
* Fixed/Disabled sites. Update requirements.txt

fixed_sites: AllRecipes, Linktree, CreativeMarket, ImgInn, Shutterstock, Contently

disabled_sites: Forums.ea.com. CrunchyRoll, Windy, MetaCritic, InfosecInstitute, Armchairgm.fandom.com, Bleach.fandom.com

Update requirements to prevent dependency conflicts.

* Update requirements.txt

Update requirements.txt to prevent dependency conflicts

* Update requirements.txt

* Update sites.md

* fixed_sites: Armchairgm.fandom.com, Bleach.fandom.com, Battleraprus. disabled_sites: MicrosoftTechNet, club.cnews.ru, Scorcher

* fixed_sites: Armchairgm.fandom.com, Bleach.fandom.com, Battleraprus. disabled_sites: MicrosoftTechNet, club.cnews.ru, Scorcher

* fixed 2 sites, disabled 22 sites, and added 1 site

* fixed 3 sites, disabled 28, added 4 sites

* update sites.md

* Added 2 more sites

* fixed 3 sites, disabled 3 sites, added 1 site

* fix Twitch. Update snapcraft.yaml. Add pyproject.toml. Remove setup.py, requirements.txt, test-requirements.txt, as they are already specified in pyproject.toml

* fix Twitch. Update snapcraft.yaml. Add pyproject.toml. Remove setup.py, requirements.txt, test-requirements.txt, as they are already specified in pyproject.toml

* fix Twitch. Update snapcraft.yaml. Add pyproject.toml. Remove setup.py, requirements.txt, test-requirements.txt, as they are already specified in pyproject.toml

* fix Twitch. Update snapcraft.yaml. Add pyproject.toml. Remove setup.py, requirements.txt, test-requirements.txt, as they are already specified in pyproject.toml

* Update sites.md

* fix Twitch. Update snapcraft.yaml. Add pyproject.toml. Remove setup.py, requirements.txt, test-requirements.txt, as they are already specified in pyproject.toml

* Update sites.md

* fix forums.drom.ru

* Add EduGeek

* Add EduGeek

* Update python-package.yml

Fix dependency installation

* Update python-package.yml

* Update python-package.yml
 2024-05-24 14:51:27 +02:00
Richard Mwewa 9399737ee6 Fixed 4 sites, added 6 sites, disabled 27 sites (#1536) 
* Fixed/Disabled sites. Update requirements.txt

fixed_sites: AllRecipes, Linktree, CreativeMarket, ImgInn, Shutterstock, Contently

disabled_sites: Forums.ea.com. CrunchyRoll, Windy, MetaCritic, InfosecInstitute, Armchairgm.fandom.com, Bleach.fandom.com

Update requirements to prevent dependency conflicts.

* Update requirements.txt

Update requirements.txt to prevent dependency conflicts

* Update requirements.txt

* Update sites.md

* fixed_sites: Armchairgm.fandom.com, Bleach.fandom.com, Battleraprus. disabled_sites: MicrosoftTechNet, club.cnews.ru, Scorcher

* fixed_sites: Armchairgm.fandom.com, Bleach.fandom.com, Battleraprus. disabled_sites: MicrosoftTechNet, club.cnews.ru, Scorcher

* fixed 2 sites, disabled 22 sites, and added 1 site

* fixed 3 sites, disabled 28, added 4 sites

* update sites.md

* Added 2 more sites
 2024-05-18 01:50:05 +02:00
Richard Mwewa f7f77e587c Fixed/Disabled sites. Update requirements.txt (#1517) 
* Fixed/Disabled sites. Update requirements.txt

fixed_sites: AllRecipes, Linktree, CreativeMarket, ImgInn, Shutterstock, Contently

disabled_sites: Forums.ea.com. CrunchyRoll, Windy, MetaCritic, InfosecInstitute, Armchairgm.fandom.com, Bleach.fandom.com

Update requirements to prevent dependency conflicts.

* Update requirements.txt

Update requirements.txt to prevent dependency conflicts

* Update requirements.txt

* Update sites.md

* fixed_sites: Armchairgm.fandom.com, Bleach.fandom.com, Battleraprus. disabled_sites: MicrosoftTechNet, club.cnews.ru, Scorcher

* fixed_sites: Armchairgm.fandom.com, Bleach.fandom.com, Battleraprus. disabled_sites: MicrosoftTechNet, club.cnews.ru, Scorcher
 2024-05-14 15:11:17 +02:00
h3x 46b13b4f23 fix reddit (#1296) 2023-11-26 18:38:06 +01:00
weekend sorrow f1ea12d731 Updating site checkers, disabling suspended sites (#1266) 
* Fixing checks for broken sites and repairing the ones that were changed

* little tweaks

* little tweaks

---------

Co-authored-by: Weekrow <somewherelse@yandex.ru>
 2023-10-27 21:43:45 +02:00
Soxoj 83d5740096 Tests fixes + last updates (#1228) 
* Some sites fixed & cloudflare detection

* Fixed issue with tests

* Updates GitHub test workflow and sites data
 2023-10-15 21:07:21 +02:00
Soxoj 90599ea3c2 Some sites fixed & cloudflare detection (#1178) 2023-09-09 20:58:01 +02:00
Soxoj c5c78b2a66 Sites fixes 250823 (#1149) 
* Additionally fixed sites, win32 build fix

* Fixed and disabled some sites (again)
 2023-08-24 10:51:24 +02:00