mirror of
https://github.com/soxoj/maigret.git
synced 2026-05-16 19:35:38 +00:00
Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Soxoj
|
ceed9aa9cc | ||
|
Soxoj
|
51a5169987 | ||
|
Soxoj
|
3e77c13743 | ||
|
Soxoj
|
c5885331d6 |
+90
-1
@@ -1,5 +1,94 @@
|
||||
# Changelog
|
||||
|
||||
## [0.6.1] - 2026-05-15
|
||||
|
||||
## What's Changed
|
||||
* build(deps): bump pypdf from 6.9.2 to 6.10.0 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2512
|
||||
* Fix duplicate attribute initialization in SimpleAiohttpChecker.__init__ by @MichaelMVS in https://github.com/soxoj/maigret/pull/2513
|
||||
* Support Python 3.14 in tests by @soxoj in https://github.com/soxoj/maigret/pull/2515
|
||||
* build(deps-dev): bump tuna from 0.5.11 to 0.5.13 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2516
|
||||
* build(deps): bump lxml from 6.0.3 to 6.0.4 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2519
|
||||
* build(deps): bump chardet from 7.4.1 to 7.4.2 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2517
|
||||
* build(deps-dev): bump mypy from 1.20.0 to 1.20.1 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2518
|
||||
* build(deps): bump pillow from 12.1.1 to 12.2.0 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2520
|
||||
* build(deps): bump chardet from 7.4.2 to 7.4.3 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2521
|
||||
* build(deps): bump pypdf from 6.10.0 to 6.10.2 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2527
|
||||
* Checks fixes by @soxoj in https://github.com/soxoj/maigret/pull/2528
|
||||
* Update of Readme and documentation by @soxoj in https://github.com/soxoj/maigret/pull/2514
|
||||
* build(deps): bump lxml from 6.0.4 to 6.1.0 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2533
|
||||
* Fix site checks: recover 6 CF sites via tls_fingerprint, 500px GraphQ… by @soxoj in https://github.com/soxoj/maigret/pull/2535
|
||||
* fix site checks: 14 sites → ip_reputation, 7 disabled, 5 dead deleted by @soxoj in https://github.com/soxoj/maigret/pull/2536
|
||||
* Fix site checks: 4 fixed, 14 → ip_reputation, 8 disabled, 5 dead deleted by @soxoj in https://github.com/soxoj/maigret/pull/2537
|
||||
* Fix site checks: 3 fixed, 2 → ip_reputation, 7 disabled, 1 dead deleted by @soxoj in https://github.com/soxoj/maigret/pull/2539
|
||||
* Add 3 crypto sites (Polymarket, Zora, Revolut.me), added crypto inves… by @soxoj in https://github.com/soxoj/maigret/pull/2538
|
||||
* Automated Sites List Update by @github-actions[bot] in https://github.com/soxoj/maigret/pull/2541
|
||||
* Fix site checks: 3 fixed, 2 → ip_reputation, 7 disabled, 1 dead deleted by @soxoj in https://github.com/soxoj/maigret/pull/2543
|
||||
* Automated Sites List Update by @github-actions[bot] in https://github.com/soxoj/maigret/pull/2545
|
||||
* Add OnlyFans with activation mechanism; updated site ranks by @soxoj in https://github.com/soxoj/maigret/pull/2546
|
||||
* build(deps-dev): bump mypy from 1.20.1 to 1.20.2 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2547
|
||||
* build(deps): bump idna from 3.11 to 3.12 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2548
|
||||
* Fix site checks: 3 → ip_reputation, 10 fixed, 6 disabled, 2 dead deleted by @soxoj in https://github.com/soxoj/maigret/pull/2549
|
||||
* Fix site checks: 12 fixed, 19 disabled; add new protection tags by @soxoj in https://github.com/soxoj/maigret/pull/2550
|
||||
* build(deps): bump certifi from 2026.2.25 to 2026.4.22 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2552
|
||||
* AI mode by @soxoj in https://github.com/soxoj/maigret/pull/2529
|
||||
* Fix site checks: 4 → ip_reputation, 9 fixed, 16 disabled, 3 dead dele… by @soxoj in https://github.com/soxoj/maigret/pull/2555
|
||||
* Fix Google Cloud Shell launch by @soxoj in https://github.com/soxoj/maigret/pull/2557
|
||||
* build(deps): bump pyinstaller from 6.19.0 to 6.20.0 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2554
|
||||
* build(deps): bump idna from 3.12 to 3.13 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2553
|
||||
* test: loosen executor timing upper bounds for slower CI by @juliosuas in https://github.com/soxoj/maigret/pull/2558
|
||||
* Fix site checks: 5 fixed; readme fix by @soxoj in https://github.com/soxoj/maigret/pull/2562
|
||||
* Add Docker web image with multi-stage building by @soxoj in https://github.com/soxoj/maigret/pull/2564
|
||||
* Fix site checks: 7 fixed, 1 disabled by @soxoj in https://github.com/soxoj/maigret/pull/2565
|
||||
* Fix site checks: 5 fixed, 4 disabled; fix UA leak bug by @soxoj in https://github.com/soxoj/maigret/pull/2569
|
||||
* build(deps): bump arabic-reshaper from 3.0.0 to 3.0.1 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2573
|
||||
* Add site checks: 18 new sites by @soxoj in https://github.com/soxoj/maigret/pull/2575
|
||||
* Automated Sites List Update by @github-actions[bot] in https://github.com/soxoj/maigret/pull/2576
|
||||
* build(deps): bump reportlab from 4.4.10 to 4.5.0 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2578
|
||||
* Fix ID extraction crash when regex groups are optional by @egrezeli in https://github.com/soxoj/maigret/pull/2572
|
||||
* Update CONTRIBUTING.md with instructions for developers by @soxoj in https://github.com/soxoj/maigret/pull/2589
|
||||
* Fix outdated Google Colab setup and dependency installation by @SayanDey322 in https://github.com/soxoj/maigret/pull/2591
|
||||
* fix: disable RomanticCollection check by @juliosuas in https://github.com/soxoj/maigret/pull/2588
|
||||
* docs: add Simplified Chinese (zh-CN) README translation by @whtis in https://github.com/soxoj/maigret/pull/2606
|
||||
* Automated Sites List Update by @github-actions[bot] in https://github.com/soxoj/maigret/pull/2607
|
||||
* Improve startup error message for missing dependencies by @SayanDey322 in https://github.com/soxoj/maigret/pull/2593
|
||||
* Modernize python package workflow by @SayanDey322 in https://github.com/soxoj/maigret/pull/2594
|
||||
* Fix site checks: 8 → ip_reputation, 6 fixed, 9 disabled, 1 dead deleted by @soxoj in https://github.com/soxoj/maigret/pull/2611
|
||||
* Reddit fix by @soxoj in https://github.com/soxoj/maigret/pull/2614
|
||||
* Automated Sites List Update by @github-actions[bot] in https://github.com/soxoj/maigret/pull/2615
|
||||
* Fix site checks: 7 fixed, 1 disabled, 1 dead deleted by @soxoj in https://github.com/soxoj/maigret/pull/2616
|
||||
* Fixed duplicates of YouTube and Periscope by @soxoj in https://github.com/soxoj/maigret/pull/2618
|
||||
* Fix network graph height to be viewport-responsive instead of fixed 750px by @SayanDey322 in https://github.com/soxoj/maigret/pull/2590
|
||||
* Add web interface tests by @soxoj in https://github.com/soxoj/maigret/pull/2619
|
||||
* refactor:reduces the cognitive complexity of get_ai_analysis by @odanilosalve in https://github.com/soxoj/maigret/pull/2581
|
||||
* AI mode documentation by @soxoj in https://github.com/soxoj/maigret/pull/2620
|
||||
* build(deps): bump python-bidi from 0.6.7 to 0.6.9 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2622
|
||||
* build(deps-dev): bump mypy from 1.20.2 to 2.0.0 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2625
|
||||
* Cloudflare bypass webgate by @soxoj in https://github.com/soxoj/maigret/pull/2628
|
||||
* Fix context field using class instead of instance in error handling by @disappear00 in https://github.com/soxoj/maigret/pull/2627
|
||||
* Add test for CheckError bug by @soxoj in https://github.com/soxoj/maigret/pull/2631
|
||||
* Update download badge links in README.md by @soxoj in https://github.com/soxoj/maigret/pull/2636
|
||||
* fix(security): harden /reports path containment via send_from_directory by @aaronjmars in https://github.com/soxoj/maigret/pull/2635
|
||||
* build(deps-dev): bump coverage from 7.13.5 to 7.14.0 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2638
|
||||
* build(deps): bump idna from 3.13 to 3.14 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2639
|
||||
* Update links to the community Telegram bot by @soxoj in https://github.com/soxoj/maigret/pull/2641
|
||||
* build(deps): bump urllib3 from 2.6.3 to 2.7.0 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2642
|
||||
* build(deps-dev): bump mypy from 2.0.0 to 2.1.0 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2644
|
||||
* Refresh stale Duolingo usernameClaimed sample (blue → duolingo) by @razbenya in https://github.com/soxoj/maigret/pull/2650
|
||||
* Fix linktr.ee detector (status_code, not stale message check) by @razbenya in https://github.com/soxoj/maigret/pull/2649
|
||||
* Apply --proxy to CurlCffiChecker (tls_fingerprint sites) by @razbenya in https://github.com/soxoj/maigret/pull/2648
|
||||
* Refresh stale Gravatar usernameClaimed sample (blue → automattic) by @razbenya in https://github.com/soxoj/maigret/pull/2651
|
||||
* Add regression tests for CurlCffiChecker proxy forwarding (#2648 follow-up) by @razbenya in https://github.com/soxoj/maigret/pull/2652
|
||||
* build(deps): bump idna from 3.14 to 3.15 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2647
|
||||
* build(deps): bump reportlab from 4.5.0 to 4.5.1 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2645
|
||||
* build(deps): bump requests from 2.33.1 to 2.34.1 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2656
|
||||
* build(deps-dev): bump pytest-rerunfailures from 16.1 to 16.2 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2654
|
||||
* build(deps): bump python-bidi from 0.6.9 to 0.6.10 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2655
|
||||
* Make xhtml2pdf optional, fix install on Linux without libcairo by @soxoj in https://github.com/soxoj/maigret/pull/2659
|
||||
* build(deps): bump requests from 2.34.1 to 2.34.2 by @dependabot[bot] in https://github.com/soxoj/maigret/pull/2658
|
||||
* Fix site checks: 2 fixed, 3 disabled; add Faceit; fix utils import by @soxoj in https://github.com/soxoj/maigret/pull/2660
|
||||
|
||||
**Full Changelog**: https://github.com/soxoj/maigret/compare/v0.6.0...v0.6.1
|
||||
|
||||
## [0.6.0] - 2025-04-10
|
||||
|
||||
## What's Changed
|
||||
@@ -778,4 +867,4 @@
|
||||
## [0.1.1] - 2020-12-05 [YANKED]
|
||||
|
||||
## [0.1.0] - 2020-12-05
|
||||
* initial release
|
||||
* initial release
|
||||
|
||||
+22
-9
@@ -51,19 +51,32 @@ pip install --upgrade certifi
|
||||
|
||||
If you are behind a corporate proxy, set `HTTPS_PROXY` / `HTTP_PROXY` environment variables and pass `--proxy "$HTTPS_PROXY"` so Maigret uses the same route.
|
||||
|
||||
## ".onion / .i2p sites are skipped"
|
||||
## Running over Tor, I2P, or Tails OS
|
||||
|
||||
These sites only load through the matching gateway. Start your Tor or I2P daemon first, then:
|
||||
Two different goals, two different flags:
|
||||
|
||||
```bash
|
||||
# Tor
|
||||
maigret user --tor-proxy socks5://127.0.0.1:9050
|
||||
- **Route only `.onion` / `.i2p` sites through their gateway** (clearweb checks still use your direct connection). Use `--tor-proxy` / `--i2p-proxy`:
|
||||
```bash
|
||||
maigret user --tor-proxy socks5://127.0.0.1:9050 # only .onion goes via Tor
|
||||
maigret user --i2p-proxy http://127.0.0.1:4444 # only .i2p goes via I2P
|
||||
```
|
||||
Without these flags, `.onion` / `.i2p` sites are silently skipped.
|
||||
|
||||
# I2P
|
||||
maigret user --i2p-proxy http://127.0.0.1:4444
|
||||
```
|
||||
- **Route the whole run through Tor / a proxy** (e.g. on Tails OS, or to anonymise the scan). Use `--proxy`:
|
||||
```bash
|
||||
# system tor daemon (apt install tor, Tails)
|
||||
maigret user --proxy socks5://127.0.0.1:9050 --timeout 60 --retries 2
|
||||
|
||||
Maigret does not launch or manage these daemons — they must already be running.
|
||||
# Tor Browser bundle (different SOCKS port!)
|
||||
maigret user --proxy socks5://127.0.0.1:9150 --timeout 60 --retries 2
|
||||
```
|
||||
Most public WAFs block Tor exits, so expect more UNKNOWNs over Tor than on a residential line — this is the cost of anonymity, not a bug. Raising `--timeout` to 60 and adding `--retries 2` materially reduces noise.
|
||||
|
||||
On Tails, `torsocks maigret …` / `torify maigret …` do **not** work — Maigret's HTTP client bypasses libc, so the wrapper has no effect. Use `--proxy` instead. To install Maigret over Tor: `torsocks pip install --user maigret`.
|
||||
|
||||
Maigret does not launch or manage Tor / I2P daemons — they must already be running.
|
||||
|
||||
For the full walkthrough (Tor Browser vs system `tor` ports, Tails persistence, reports paths), see the [Tor, I2P, and proxies](https://maigret.readthedocs.io/en/latest/tor-and-proxies.html) page on readthedocs.
|
||||
|
||||
## "The PDF / XMind / HTML report looks wrong"
|
||||
|
||||
|
||||
@@ -63,6 +63,29 @@ from slow sites. On the other hand, this may cause a long delay to
|
||||
gather all results. The choice of the right timeout should be carried
|
||||
out taking into account the bandwidth of the Internet connection.
|
||||
|
||||
Network and proxy options
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
``--proxy PROXY_URL`` / ``-p PROXY_URL`` - Route **every** check through
|
||||
the given HTTP or SOCKS proxy. Example: ``socks5://127.0.0.1:1080``,
|
||||
``http://user:pass@proxy.example:3128``. This is the flag to use for
|
||||
routing the whole run through Tor (``--proxy socks5://127.0.0.1:9050``),
|
||||
a residential proxy, or any corporate gateway. No default.
|
||||
|
||||
``--tor-proxy TOR_PROXY_URL`` - Gateway used **only** for ``.onion``
|
||||
sites in the database **(default: socks5://127.0.0.1:9050)**. Clearweb
|
||||
sites are unaffected — for them Maigret uses your direct connection or
|
||||
``--proxy`` if you set one. Without this flag, ``.onion`` sites are
|
||||
silently skipped.
|
||||
|
||||
``--i2p-proxy I2P_PROXY_URL`` - Gateway used **only** for ``.i2p``
|
||||
sites in the database **(default: http://127.0.0.1:4444)**. Same
|
||||
"only matching protocol" rule as ``--tor-proxy``.
|
||||
|
||||
Maigret does not start the Tor or I2P daemon for you — launch it first.
|
||||
For a full walkthrough (Tor Browser vs system ``tor`` port numbers,
|
||||
Tails OS recipe, timeout/retry tuning), see :doc:`tor-and-proxies`.
|
||||
|
||||
``--cookies-jar-file`` - File with custom cookies in Netscape format
|
||||
(aka cookies.txt). You can install an extension to your browser to
|
||||
download own cookies (`Chrome <https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid>`_, `Firefox <https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/>`_).
|
||||
|
||||
+2
-2
@@ -6,8 +6,8 @@ project = 'Maigret'
|
||||
copyright = '2025, soxoj'
|
||||
author = 'soxoj'
|
||||
|
||||
release = '0.5.0'
|
||||
version = '0.5'
|
||||
release = '0.6.1'
|
||||
version = '0.6'
|
||||
|
||||
# -- General configuration
|
||||
|
||||
|
||||
@@ -283,12 +283,21 @@ PyPi package.
|
||||
|
||||
git checkout -b 0.4.0
|
||||
|
||||
2. Update Maigret version in three files manually:
|
||||
2. Update Maigret version in four files manually. **All four must be in
|
||||
sync** — the previous bump missed ``docs/source/conf.py`` and
|
||||
``snapcraft.yaml`` and they fell behind by a release.
|
||||
|
||||
- pyproject.toml
|
||||
- maigret/__version__.py
|
||||
- docs/source/conf.py
|
||||
- snapcraft.yaml
|
||||
- ``pyproject.toml`` — single line ``version = "X.Y.Z"`` under
|
||||
``[tool.poetry]``.
|
||||
- ``maigret/__version__.py`` — single line ``__version__ = 'X.Y.Z'``.
|
||||
- ``docs/source/conf.py`` — **two** Sphinx fields. ``release`` is the
|
||||
full version (``'X.Y.Z'``); ``version`` is the short ``major.minor``
|
||||
(``'X.Y'``, **without** the patch number). Update **both**.
|
||||
- ``snapcraft.yaml`` — single line ``version: X.Y.Z`` (no quotes, no
|
||||
``v`` prefix).
|
||||
|
||||
After editing, sanity-check with ``grep -rE '0\.5\.|0\.6\.|<old>'`` to
|
||||
catch any straggler reference.
|
||||
|
||||
3. Create a new empty text section in the beginning of the file `CHANGELOG.md` with a current date:
|
||||
|
||||
|
||||
@@ -30,6 +30,7 @@ You may be interested in:
|
||||
- :doc:`Command line options <command-line-options>`
|
||||
- :doc:`Features list <features>`
|
||||
- :doc:`Library usage <library-usage>`
|
||||
- :doc:`Tor, I2P, and proxies <tor-and-proxies>`
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
@@ -40,13 +41,19 @@ You may be interested in:
|
||||
usage-examples
|
||||
command-line-options
|
||||
features
|
||||
library-usage
|
||||
philosophy
|
||||
supported-identifier-types
|
||||
tags
|
||||
settings
|
||||
development
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:caption: Advanced usage
|
||||
|
||||
library-usage
|
||||
settings
|
||||
tor-and-proxies
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:caption: Use cases
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
.. _tor-and-proxies:
|
||||
|
||||
Tor, I2P, and proxies
|
||||
=====================
|
||||
|
||||
Maigret can route checks through an HTTP/SOCKS proxy, the Tor network, or I2P. Three CLI flags cover three distinct goals — knowing which one you need is the most common stumbling block.
|
||||
|
||||
``--proxy`` vs ``--tor-proxy`` (and ``--i2p-proxy``)
|
||||
----------------------------------------------------
|
||||
|
||||
The most-asked question (see `issue #544 <https://github.com/soxoj/maigret/issues/544>`_):
|
||||
|
||||
- **You want every check to go through Tor** (e.g. you're on Tails OS, or behind a country-level block, or your IP is rate-limited). → Use ``--proxy``, pointing at your Tor SOCKS port:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
maigret <username> --proxy socks5://127.0.0.1:9050
|
||||
|
||||
- **You want to reach ``.onion`` sites in the Maigret database**, while the rest of the run still uses your normal connection. → Use ``--tor-proxy``:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
maigret <username> --tor-proxy socks5://127.0.0.1:9050
|
||||
|
||||
``--tor-proxy`` is **only** consulted for sites whose ``url`` is a ``.onion`` host. For every other site Maigret uses your direct connection (or ``--proxy`` if set). Without ``--tor-proxy``, ``.onion`` sites are silently skipped.
|
||||
|
||||
The same split applies to ``--i2p-proxy``: it is consulted only for ``.i2p`` hosts, never for clearweb sites.
|
||||
|
||||
Defaults: ``--tor-proxy`` defaults to ``socks5://127.0.0.1:9050`` and ``--i2p-proxy`` to ``http://127.0.0.1:4444``. ``--proxy`` has no default. Maigret does **not** launch ``tor`` or an I2P router for you — start the daemon first.
|
||||
|
||||
Tor Browser vs system ``tor``: port numbers
|
||||
-------------------------------------------
|
||||
|
||||
The SOCKS port differs by Tor installation:
|
||||
|
||||
- **System ``tor`` daemon** (``apt install tor``, ``brew install tor``, Tails) listens on ``9050``.
|
||||
- **Tor Browser bundle** ships its own ``tor`` listening on ``9150``.
|
||||
|
||||
If a connection refuses, try the other port:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# system tor
|
||||
maigret <username> --proxy socks5://127.0.0.1:9050
|
||||
|
||||
# Tor Browser running in the background
|
||||
maigret <username> --proxy socks5://127.0.0.1:9150
|
||||
|
||||
A note on results over Tor
|
||||
--------------------------
|
||||
|
||||
Most public WAFs (Cloudflare, DDoS-Guard, AWS WAF, Akamai) block Tor exit nodes by default — usually more aggressively than they block datacenter IPs. A Tor run typically produces **more UNKNOWNs and fewer CLAIMEDs** than the same run from a residential connection. This is not a bug in Maigret; it is the cost of anonymity.
|
||||
|
||||
Recommended flags for a Tor run:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
maigret <username> --proxy socks5://127.0.0.1:9050 --timeout 60 --retries 2
|
||||
|
||||
- ``--timeout 60`` — Tor circuits add 1–3 seconds per request; the default 30 s causes spurious timeouts.
|
||||
- ``--retries 2`` — retries cover transient circuit failures, which are common on Tor.
|
||||
- Optional ``-n 20`` — lowering concurrency (default 100) reduces the chance of exits rate-limiting you.
|
||||
|
||||
If you mainly need to bypass WAFs (rather than to remain anonymous), a residential proxy will usually outperform Tor by a wide margin. See the **"Lots of sites fail / timeout / return 403"** section in `TROUBLESHOOTING.md <https://github.com/soxoj/maigret/blob/main/TROUBLESHOOTING.md>`_.
|
||||
|
||||
Running on Tails OS
|
||||
-------------------
|
||||
|
||||
Tails forces every outbound connection through Tor at the network layer. Maigret needs no special configuration to comply — pointing ``--proxy`` at the Tails Tor daemon is enough:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
maigret <username> --proxy socks5://127.0.0.1:9050 --timeout 60
|
||||
|
||||
Things that are **not** needed:
|
||||
|
||||
- ``torsocks maigret …`` and ``torify maigret …`` — these wrap libc socket calls, but Maigret's HTTP client (``aiohttp`` / ``curl_cffi``) bypasses libc for network I/O, so the wrapper has no effect. Use ``--proxy`` instead.
|
||||
- ``--tor-proxy`` — on Tails, *everything* must go via Tor (the OS enforces this), so the niche "only .onion via Tor" mode that ``--tor-proxy`` provides does not apply.
|
||||
|
||||
Installation over Tor on Tails
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
``pip`` itself does not know about Tor; on Tails you need ``torsocks`` to wrap it:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
torsocks pip install --user maigret
|
||||
|
||||
After install, the binary lands in ``~/.local/bin/maigret``. If ``maigret: command not found``, either add ``~/.local/bin`` to ``PATH`` or invoke it as ``python3 -m maigret <username>``.
|
||||
|
||||
Persisting Maigret across Tails sessions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Tails wipes ``~/.local/`` on reboot unless you configure the Persistent Storage to keep it. This is Tails configuration, not Maigret configuration — see the official Tails docs:
|
||||
|
||||
- `Persistent Storage on Tails <https://tails.boum.org/doc/persistent_storage/>`_
|
||||
- `Configuring Persistent Storage features <https://tails.boum.org/doc/persistent_storage/configure/>`_
|
||||
|
||||
A step-by-step recipe contributed by a user (persisting ``~/.local/lib/python3.9`` and ``~/.local/bin`` and patching ``.bashrc``) is in `issue #544 <https://github.com/soxoj/maigret/issues/544#issuecomment-1356469171>`_. Treat it as a starting point: the Python version and Tails internals change between Tails releases.
|
||||
|
||||
Reports on Tails — where to save them
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The default ``reports/`` directory lives next to the working directory and is wiped with the amnesiac session. To save reports somewhere persistent, either pass ``-fo``:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
maigret <username> --html -fo "/home/amnesia/Persistent/maigret-reports"
|
||||
|
||||
or set ``"reports_path"`` in your ``settings.json`` to a persistent path. See :doc:`settings`.
|
||||
|
||||
Programmatic equivalents (Python library)
|
||||
-----------------------------------------
|
||||
|
||||
The same options are available through the Python API. See :doc:`library-usage` — the relevant keyword arguments are ``proxy=``, ``tor_proxy=`` and ``i2p_proxy=``, accepting the same URL formats as the CLI flags.
|
||||
|
||||
See also
|
||||
--------
|
||||
|
||||
- :doc:`command-line-options` — full reference for the three flags.
|
||||
- `TROUBLESHOOTING.md <https://github.com/soxoj/maigret/blob/main/TROUBLESHOOTING.md>`_ — quick recipes for ``.onion`` / I2P sites and for WAF-induced 403s.
|
||||
- :doc:`library-usage` — proxy options for embedded use.
|
||||
@@ -1,3 +1,3 @@
|
||||
"""Maigret version file"""
|
||||
|
||||
__version__ = '0.6.0'
|
||||
__version__ = '0.6.1'
|
||||
|
||||
+13
-6
@@ -31,7 +31,7 @@ from .executors import AsyncioQueueGeneratorExecutor
|
||||
from .result import MaigretCheckResult, MaigretCheckStatus
|
||||
from .sites import MaigretDatabase, MaigretSite
|
||||
from .types import QueryOptions, QueryResultWrapper
|
||||
from .utils import ascii_data_display, get_random_user_agent
|
||||
from .utils import ascii_data_display, get_random_user_agent, is_plausible_username
|
||||
|
||||
|
||||
SUPPORTED_IDS = (
|
||||
@@ -639,7 +639,6 @@ def process_site_result(
|
||||
|
||||
html_text, status_code, check_error = response
|
||||
|
||||
# TODO: add elapsed request time counting
|
||||
response_time = None
|
||||
|
||||
if logger.level == logging.DEBUG:
|
||||
@@ -673,7 +672,6 @@ def process_site_result(
|
||||
f"Failed activation {method} for site {site.name}: {str(e)}",
|
||||
exc_info=True,
|
||||
)
|
||||
# TODO: temporary check error
|
||||
|
||||
site_name = site.pretty_name
|
||||
# presense flags
|
||||
@@ -1296,7 +1294,6 @@ async def site_self_check(
|
||||
)
|
||||
|
||||
# don't disable entries with other ids types
|
||||
# TODO: make normal checking
|
||||
if site.name not in results_dict:
|
||||
logger.info(results_dict)
|
||||
changes["issues"].append(f"Site {site.name} not in results (wrong id_type?)")
|
||||
@@ -1525,13 +1522,23 @@ def parse_usernames(extracted_ids_data, logger) -> Dict:
|
||||
new_usernames = {}
|
||||
for k, v in extracted_ids_data.items():
|
||||
if "username" in k and not "usernames" in k:
|
||||
new_usernames[v] = "username"
|
||||
if is_plausible_username(v):
|
||||
new_usernames[v] = "username"
|
||||
else:
|
||||
logger.debug(
|
||||
f"Rejected non-username value extracted under key {k!r}: {v!r}"
|
||||
)
|
||||
elif "usernames" in k:
|
||||
try:
|
||||
tree = ast.literal_eval(v)
|
||||
if isinstance(tree, list):
|
||||
for n in tree:
|
||||
new_usernames[n] = "username"
|
||||
if is_plausible_username(n):
|
||||
new_usernames[n] = "username"
|
||||
else:
|
||||
logger.debug(
|
||||
f"Rejected non-username item from list under key {k!r}: {n!r}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(e)
|
||||
if k in SUPPORTED_IDS:
|
||||
|
||||
@@ -77,7 +77,6 @@ ERRORS_TYPES = {
|
||||
'Connecting failure': 'Try to decrease number of parallel connections (e.g. -n 10)',
|
||||
}
|
||||
|
||||
# TODO: checking for reason
|
||||
ERRORS_REASONS = {
|
||||
'Login required': 'Add authorization cookies through `--cookies-jar-file` (see cookies.txt)',
|
||||
}
|
||||
|
||||
+13
-3
@@ -55,7 +55,7 @@ from .report import (
|
||||
from .sites import MaigretDatabase
|
||||
from .submit import Submitter
|
||||
from .types import QueryResultWrapper
|
||||
from .utils import get_dict_ascii_tree
|
||||
from .utils import get_dict_ascii_tree, is_plausible_username
|
||||
from .settings import Settings
|
||||
from .permutator import Permute
|
||||
|
||||
@@ -85,13 +85,23 @@ def extract_ids_from_page(url, logger, timeout=5) -> dict:
|
||||
for k, v in info.items():
|
||||
# TODO: merge with the same functionality in checking module
|
||||
if 'username' in k and not 'usernames' in k:
|
||||
results[v] = 'username'
|
||||
if is_plausible_username(v):
|
||||
results[v] = 'username'
|
||||
else:
|
||||
logger.debug(
|
||||
f"Rejected non-username value extracted under key {k!r}: {v!r}"
|
||||
)
|
||||
elif 'usernames' in k:
|
||||
try:
|
||||
tree = ast.literal_eval(v)
|
||||
if isinstance(tree, list):
|
||||
for n in tree:
|
||||
results[n] = 'username'
|
||||
if is_plausible_username(n):
|
||||
results[n] = 'username'
|
||||
else:
|
||||
logger.debug(
|
||||
f"Rejected non-username item from list under key {k!r}: {n!r}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(e)
|
||||
if k in SUPPORTED_IDS:
|
||||
|
||||
@@ -516,7 +516,6 @@ def generate_report_context(username_results: list):
|
||||
tag = pycountry.countries.search_fuzzy(v)[
|
||||
0
|
||||
].alpha_2.lower() # type: ignore[attr-defined]
|
||||
# TODO: move countries to another struct
|
||||
tags[tag] = tags.get(tag, 0) + 1
|
||||
except Exception as e:
|
||||
logging.debug(
|
||||
@@ -568,7 +567,6 @@ def generate_report_context(username_results: list):
|
||||
|
||||
return {
|
||||
"username": first_username,
|
||||
# TODO: return brief list
|
||||
"brief": brief,
|
||||
"results": username_results,
|
||||
"first_seen": first_seen,
|
||||
|
||||
@@ -57,7 +57,8 @@
|
||||
"\"routePath\":null"
|
||||
],
|
||||
"errors": {
|
||||
"Login • Instagram": "Login required"
|
||||
"Login • Instagram": "Login required",
|
||||
"\"routePath\":\"\\/\"": "Login required (rate-limited or session blocked)"
|
||||
},
|
||||
"alexaRank": 4,
|
||||
"urlMain": "https://www.instagram.com/",
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
{
|
||||
"version": 1,
|
||||
"updated_at": "2026-05-15T16:12:58Z",
|
||||
"updated_at": "2026-05-16T16:00:20Z",
|
||||
"sites_count": 3155,
|
||||
"min_maigret_version": "0.6.0",
|
||||
"data_sha256": "df2ab3dbc96bdcdc8aa4e9da485df75ce6c3274814080f00a35e89f7f43783e1",
|
||||
"min_maigret_version": "0.6.1",
|
||||
"data_sha256": "0997b68c05eedb6e714432ed79580688d4923c56ef1ebf46db69b90039ef00d7",
|
||||
"data_url": "https://raw.githubusercontent.com/soxoj/maigret/main/maigret/resources/data.json"
|
||||
}
|
||||
@@ -127,3 +127,29 @@ def get_match_ratio(base_strs: list):
|
||||
|
||||
def generate_random_username():
|
||||
return ''.join(random.choices(string.ascii_lowercase, k=10))
|
||||
|
||||
|
||||
def is_plausible_username(value: Any) -> bool:
|
||||
"""Reject obviously non-username strings extracted from sites' identity data.
|
||||
|
||||
Extractor schemes occasionally populate fields named like ``*_username``
|
||||
with URLs (e.g. ``instagram_username`` -> ``https://instagram.com/X``) or
|
||||
emails (e.g. ``your_username`` -> ``user@example.com``). Feeding such a
|
||||
value back into a site URL template produces broken requests on every
|
||||
subsequent site, which manifests as a cascade of false errors and the
|
||||
"wrong username" symptom in #1403.
|
||||
"""
|
||||
if not isinstance(value, str):
|
||||
return False
|
||||
s = value.strip()
|
||||
if not s:
|
||||
return False
|
||||
if "://" in s or s.startswith(("http://", "https://", "www.", "//")):
|
||||
return False
|
||||
if "/" in s:
|
||||
return False
|
||||
if any(c.isspace() for c in s):
|
||||
return False
|
||||
if "@" in s and "." in s:
|
||||
return False
|
||||
return True
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
|
||||
|
||||
[tool.poetry]
|
||||
name = "maigret"
|
||||
version = "0.6.0"
|
||||
version = "0.6.1"
|
||||
description = "🕵️♂️ Collect a dossier on a person by username from thousands of sites."
|
||||
authors = ["Soxoj <soxoj@protonmail.com>"]
|
||||
readme = "README.md"
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ description: |
|
||||
|
||||
Currently supported more than 3000 sites, search is launched against 500 popular sites in descending order of popularity by default. Also supported checking of Tor sites, I2P sites, and domains (via DNS resolving).
|
||||
|
||||
version: 0.5.0
|
||||
version: 0.6.1
|
||||
license: MIT
|
||||
base: core22
|
||||
confinement: strict
|
||||
|
||||
@@ -126,6 +126,40 @@ def test_detect_error_page_ok():
|
||||
assert detect_error_page("hello world", 200, {}, ignore_403=False) is None
|
||||
|
||||
|
||||
def test_detect_error_page_instagram_login_wall():
|
||||
"""Regression for #11: when Instagram serves the login wall (typically the
|
||||
response after rate-limiting an unauthenticated client), the JSON state
|
||||
contains `"routePath":"\\/"` (root path) rather than a username route. The
|
||||
Instagram entry in data.json carries this marker in `errors` so the result
|
||||
surfaces as UNKNOWN instead of a false AVAILABLE.
|
||||
"""
|
||||
instagram_errors = {
|
||||
"Login • Instagram": "Login required",
|
||||
'"routePath":"\\/"': "Login required (rate-limited or session blocked)",
|
||||
}
|
||||
login_wall_html = '...{"routePath":"\\/"},"timeSpent":...'
|
||||
err = detect_error_page(login_wall_html, 200, instagram_errors, ignore_403=False)
|
||||
assert err is not None
|
||||
assert err.type == "Site-specific"
|
||||
assert "rate-limited" in err.desc
|
||||
|
||||
|
||||
def test_detect_error_page_instagram_marker_no_false_positive_on_profile():
|
||||
"""The login-wall marker must NOT match a real profile page. On a claimed
|
||||
user page, `routePath` carries the user-route template
|
||||
(`"routePath":"\\/{username}\\/..."`); the closing-quote form
|
||||
`"routePath":"\\/"` only appears on the login wall.
|
||||
"""
|
||||
instagram_errors = {
|
||||
'"routePath":"\\/"': "Login required (rate-limited or session blocked)",
|
||||
}
|
||||
profile_html = (
|
||||
'foo,"routePath":"\\/{username}\\/{?tab}\\/{?view_type}\\/",bar'
|
||||
)
|
||||
err = detect_error_page(profile_html, 200, instagram_errors, ignore_403=False)
|
||||
assert err is None
|
||||
|
||||
|
||||
def test_parse_usernames_single_username():
|
||||
logger = Mock()
|
||||
result = parse_usernames({"profile_username": "alice"}, logger)
|
||||
@@ -146,6 +180,33 @@ def test_parse_usernames_malformed_list():
|
||||
assert logger.warning.called
|
||||
|
||||
|
||||
def test_parse_usernames_rejects_url_value():
|
||||
"""Regression for #1403: extractors sometimes return a URL under a *_username
|
||||
key; that URL must not be fed back as a candidate username."""
|
||||
logger = Mock()
|
||||
result = parse_usernames(
|
||||
{"instagram_username": "https://instagram.com/zuck"}, logger
|
||||
)
|
||||
assert result == {}
|
||||
|
||||
|
||||
def test_parse_usernames_rejects_email_value():
|
||||
"""Regression for #1403: e.g. socid_extractor's 'your_username' returns an
|
||||
email under a key matching the username heuristic."""
|
||||
logger = Mock()
|
||||
result = parse_usernames({"your_username": "alice@example.com"}, logger)
|
||||
assert result == {}
|
||||
|
||||
|
||||
def test_parse_usernames_filters_urls_inside_list():
|
||||
logger = Mock()
|
||||
result = parse_usernames(
|
||||
{"other_usernames": "['alice', 'https://example.com/bob']"}, logger
|
||||
)
|
||||
# 'alice' should survive; the URL should be dropped.
|
||||
assert result == {"alice": "username"}
|
||||
|
||||
|
||||
def test_parse_usernames_supported_id():
|
||||
logger = Mock()
|
||||
# "telegram" is in SUPPORTED_IDS per socid_extractor
|
||||
|
||||
@@ -10,6 +10,7 @@ from maigret.utils import (
|
||||
URLMatcher,
|
||||
get_dict_ascii_tree,
|
||||
get_match_ratio,
|
||||
is_plausible_username,
|
||||
)
|
||||
|
||||
|
||||
@@ -144,3 +145,52 @@ def test_get_match_ratio():
|
||||
fun = get_match_ratio(["test", "maigret", "username"])
|
||||
|
||||
assert fun("test") == 1
|
||||
|
||||
|
||||
# Regression tests for #1403 — Gravatar URL leaking into next-iteration username.
|
||||
# Extractor schemes occasionally store URLs/emails under '*_username' keys; without
|
||||
# validation these were fed back into the search loop and produced cascades of false
|
||||
# errors. See maigret/utils.py::is_plausible_username.
|
||||
def test_is_plausible_username_accepts_bare_usernames():
|
||||
assert is_plausible_username("alice")
|
||||
assert is_plausible_username("alice.bob")
|
||||
assert is_plausible_username("alice_bob-42")
|
||||
assert is_plausible_username("Алиса")
|
||||
|
||||
|
||||
def test_is_plausible_username_rejects_urls():
|
||||
assert not is_plausible_username("https://gravatar.com/alice")
|
||||
assert not is_plausible_username("http://example.com/user/alice")
|
||||
assert not is_plausible_username("//example.com/alice")
|
||||
assert not is_plausible_username("www.facebook.com/zuck")
|
||||
|
||||
|
||||
def test_is_plausible_username_accepts_http_prefixed_handles():
|
||||
"""Don't over-match: bare names that just happen to start with 'http' or 'www'
|
||||
are legitimate (e.g. the httpie CLI maintainer's handle)."""
|
||||
assert is_plausible_username("httpie")
|
||||
assert is_plausible_username("http_user")
|
||||
assert is_plausible_username("wwwsuperstar")
|
||||
|
||||
|
||||
def test_is_plausible_username_rejects_path_like():
|
||||
assert not is_plausible_username("user/alice")
|
||||
assert not is_plausible_username("alice/")
|
||||
|
||||
|
||||
def test_is_plausible_username_rejects_emails():
|
||||
assert not is_plausible_username("alice@example.com")
|
||||
assert not is_plausible_username("user@maigret.io")
|
||||
|
||||
|
||||
def test_is_plausible_username_rejects_whitespace_and_empty():
|
||||
assert not is_plausible_username("")
|
||||
assert not is_plausible_username(" ")
|
||||
assert not is_plausible_username("alice bob")
|
||||
assert not is_plausible_username("alice\nbob")
|
||||
|
||||
|
||||
def test_is_plausible_username_rejects_non_strings():
|
||||
assert not is_plausible_username(None)
|
||||
assert not is_plausible_username(42)
|
||||
assert not is_plausible_username(["alice"])
|
||||
|
||||
@@ -165,7 +165,6 @@ if __name__ == '__main__':
|
||||
sites = {site.name: site for site in sites_subset}
|
||||
engines = db.engines
|
||||
|
||||
# TODO: usernames extractors
|
||||
ok_usernames = ['alex', 'god', 'admin', 'red', 'blue', 'john']
|
||||
if args.username:
|
||||
ok_usernames = [args.username] + ok_usernames
|
||||
|
||||
Reference in New Issue
Block a user