mirror of
https://github.com/soxoj/maigret.git
synced 2026-05-16 19:35:38 +00:00
Document Tor/proxy usage, add Advanced usage docs section (closes #544)
This commit is contained in:
Soxoj
@@ -63,6 +63,29 @@ from slow sites. On the other hand, this may cause a long delay to
|
||||
gather all results. The choice of the right timeout should be carried
|
||||
out taking into account the bandwidth of the Internet connection.
|
||||
|
||||
Network and proxy options
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
``--proxy PROXY_URL`` / ``-p PROXY_URL`` - Route **every** check through
|
||||
the given HTTP or SOCKS proxy. Example: ``socks5://127.0.0.1:1080``,
|
||||
``http://user:pass@proxy.example:3128``. This is the flag to use for
|
||||
routing the whole run through Tor (``--proxy socks5://127.0.0.1:9050``),
|
||||
a residential proxy, or any corporate gateway. No default.
|
||||
|
||||
``--tor-proxy TOR_PROXY_URL`` - Gateway used **only** for ``.onion``
|
||||
sites in the database **(default: socks5://127.0.0.1:9050)**. Clearweb
|
||||
sites are unaffected — for them Maigret uses your direct connection or
|
||||
``--proxy`` if you set one. Without this flag, ``.onion`` sites are
|
||||
silently skipped.
|
||||
|
||||
``--i2p-proxy I2P_PROXY_URL`` - Gateway used **only** for ``.i2p``
|
||||
sites in the database **(default: http://127.0.0.1:4444)**. Same
|
||||
"only matching protocol" rule as ``--tor-proxy``.
|
||||
|
||||
Maigret does not start the Tor or I2P daemon for you — launch it first.
|
||||
For a full walkthrough (Tor Browser vs system ``tor`` port numbers,
|
||||
Tails OS recipe, timeout/retry tuning), see :doc:`tor-and-proxies`.
|
||||
|
||||
``--cookies-jar-file`` - File with custom cookies in Netscape format
|
||||
(aka cookies.txt). You can install an extension to your browser to
|
||||
download own cookies (`Chrome <https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid>`_, `Firefox <https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/>`_).
|
||||
|
||||
@@ -30,6 +30,7 @@ You may be interested in:
|
||||
- :doc:`Command line options <command-line-options>`
|
||||
- :doc:`Features list <features>`
|
||||
- :doc:`Library usage <library-usage>`
|
||||
- :doc:`Tor, I2P, and proxies <tor-and-proxies>`
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
@@ -40,13 +41,19 @@ You may be interested in:
|
||||
usage-examples
|
||||
command-line-options
|
||||
features
|
||||
library-usage
|
||||
philosophy
|
||||
supported-identifier-types
|
||||
tags
|
||||
settings
|
||||
development
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:caption: Advanced usage
|
||||
|
||||
library-usage
|
||||
settings
|
||||
tor-and-proxies
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:caption: Use cases
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
.. _tor-and-proxies:
|
||||
|
||||
Tor, I2P, and proxies
|
||||
=====================
|
||||
|
||||
Maigret can route checks through an HTTP/SOCKS proxy, the Tor network, or I2P. Three CLI flags cover three distinct goals — knowing which one you need is the most common stumbling block.
|
||||
|
||||
``--proxy`` vs ``--tor-proxy`` (and ``--i2p-proxy``)
|
||||
----------------------------------------------------
|
||||
|
||||
The most-asked question (see `issue #544 <https://github.com/soxoj/maigret/issues/544>`_):
|
||||
|
||||
- **You want every check to go through Tor** (e.g. you're on Tails OS, or behind a country-level block, or your IP is rate-limited). → Use ``--proxy``, pointing at your Tor SOCKS port:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
maigret <username> --proxy socks5://127.0.0.1:9050
|
||||
|
||||
- **You want to reach ``.onion`` sites in the Maigret database**, while the rest of the run still uses your normal connection. → Use ``--tor-proxy``:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
maigret <username> --tor-proxy socks5://127.0.0.1:9050
|
||||
|
||||
``--tor-proxy`` is **only** consulted for sites whose ``url`` is a ``.onion`` host. For every other site Maigret uses your direct connection (or ``--proxy`` if set). Without ``--tor-proxy``, ``.onion`` sites are silently skipped.
|
||||
|
||||
The same split applies to ``--i2p-proxy``: it is consulted only for ``.i2p`` hosts, never for clearweb sites.
|
||||
|
||||
Defaults: ``--tor-proxy`` defaults to ``socks5://127.0.0.1:9050`` and ``--i2p-proxy`` to ``http://127.0.0.1:4444``. ``--proxy`` has no default. Maigret does **not** launch ``tor`` or an I2P router for you — start the daemon first.
|
||||
|
||||
Tor Browser vs system ``tor``: port numbers
|
||||
-------------------------------------------
|
||||
|
||||
The SOCKS port differs by Tor installation:
|
||||
|
||||
- **System ``tor`` daemon** (``apt install tor``, ``brew install tor``, Tails) listens on ``9050``.
|
||||
- **Tor Browser bundle** ships its own ``tor`` listening on ``9150``.
|
||||
|
||||
If a connection refuses, try the other port:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# system tor
|
||||
maigret <username> --proxy socks5://127.0.0.1:9050
|
||||
|
||||
# Tor Browser running in the background
|
||||
maigret <username> --proxy socks5://127.0.0.1:9150
|
||||
|
||||
A note on results over Tor
|
||||
--------------------------
|
||||
|
||||
Most public WAFs (Cloudflare, DDoS-Guard, AWS WAF, Akamai) block Tor exit nodes by default — usually more aggressively than they block datacenter IPs. A Tor run typically produces **more UNKNOWNs and fewer CLAIMEDs** than the same run from a residential connection. This is not a bug in Maigret; it is the cost of anonymity.
|
||||
|
||||
Recommended flags for a Tor run:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
maigret <username> --proxy socks5://127.0.0.1:9050 --timeout 60 --retries 2
|
||||
|
||||
- ``--timeout 60`` — Tor circuits add 1–3 seconds per request; the default 30 s causes spurious timeouts.
|
||||
- ``--retries 2`` — retries cover transient circuit failures, which are common on Tor.
|
||||
- Optional ``-n 20`` — lowering concurrency (default 100) reduces the chance of exits rate-limiting you.
|
||||
|
||||
If you mainly need to bypass WAFs (rather than to remain anonymous), a residential proxy will usually outperform Tor by a wide margin. See the **"Lots of sites fail / timeout / return 403"** section in `TROUBLESHOOTING.md <https://github.com/soxoj/maigret/blob/main/TROUBLESHOOTING.md>`_.
|
||||
|
||||
Running on Tails OS
|
||||
-------------------
|
||||
|
||||
Tails forces every outbound connection through Tor at the network layer. Maigret needs no special configuration to comply — pointing ``--proxy`` at the Tails Tor daemon is enough:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
maigret <username> --proxy socks5://127.0.0.1:9050 --timeout 60
|
||||
|
||||
Things that are **not** needed:
|
||||
|
||||
- ``torsocks maigret …`` and ``torify maigret …`` — these wrap libc socket calls, but Maigret's HTTP client (``aiohttp`` / ``curl_cffi``) bypasses libc for network I/O, so the wrapper has no effect. Use ``--proxy`` instead.
|
||||
- ``--tor-proxy`` — on Tails, *everything* must go via Tor (the OS enforces this), so the niche "only .onion via Tor" mode that ``--tor-proxy`` provides does not apply.
|
||||
|
||||
Installation over Tor on Tails
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
``pip`` itself does not know about Tor; on Tails you need ``torsocks`` to wrap it:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
torsocks pip install --user maigret
|
||||
|
||||
After install, the binary lands in ``~/.local/bin/maigret``. If ``maigret: command not found``, either add ``~/.local/bin`` to ``PATH`` or invoke it as ``python3 -m maigret <username>``.
|
||||
|
||||
Persisting Maigret across Tails sessions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Tails wipes ``~/.local/`` on reboot unless you configure the Persistent Storage to keep it. This is Tails configuration, not Maigret configuration — see the official Tails docs:
|
||||
|
||||
- `Persistent Storage on Tails <https://tails.boum.org/doc/persistent_storage/>`_
|
||||
- `Configuring Persistent Storage features <https://tails.boum.org/doc/persistent_storage/configure/>`_
|
||||
|
||||
A step-by-step recipe contributed by a user (persisting ``~/.local/lib/python3.9`` and ``~/.local/bin`` and patching ``.bashrc``) is in `issue #544 <https://github.com/soxoj/maigret/issues/544#issuecomment-1356469171>`_. Treat it as a starting point: the Python version and Tails internals change between Tails releases.
|
||||
|
||||
Reports on Tails — where to save them
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The default ``reports/`` directory lives next to the working directory and is wiped with the amnesiac session. To save reports somewhere persistent, either pass ``-fo``:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
maigret <username> --html -fo "/home/amnesia/Persistent/maigret-reports"
|
||||
|
||||
or set ``"reports_path"`` in your ``settings.json`` to a persistent path. See :doc:`settings`.
|
||||
|
||||
Programmatic equivalents (Python library)
|
||||
-----------------------------------------
|
||||
|
||||
The same options are available through the Python API. See :doc:`library-usage` — the relevant keyword arguments are ``proxy=``, ``tor_proxy=`` and ``i2p_proxy=``, accepting the same URL formats as the CLI flags.
|
||||
|
||||
See also
|
||||
--------
|
||||
|
||||
- :doc:`command-line-options` — full reference for the three flags.
|
||||
- `TROUBLESHOOTING.md <https://github.com/soxoj/maigret/blob/main/TROUBLESHOOTING.md>`_ — quick recipes for ``.onion`` / I2P sites and for WAF-induced 403s.
|
||||
- :doc:`library-usage` — proxy options for embedded use.
|
||||
Reference in New Issue
Block a user