Added dev documentation, fixed some sites, removed GitHub issue links from reports (#1869)

docs/source/development.rst

@@ -3,10 +3,31 @@
 Development
 ==============
 
+Frequently Asked Questions
+------------------------- 
+
+1. Where to find the list of supported sites?
+
+The human-readable list of supported sites is available in the `sites.md <https://github.com/soxoj/maigret/blob/main/sites.md>`_ file in the repository.
+It's been generated automatically from the main JSON file with the list of supported sites.
+
+The machine-readable JSON file with the list of supported sites is available in the
+`data.json <https://github.com/soxoj/maigret/blob/main/maigret/resources/data.json>`_ file in the directory `resources`.
+
+2. Which methods to check the account presence are supported?
+
+The supported methods (``checkType`` values in ``data.json``) are:
+
+- ``message`` - the most reliable method, checks if any string from ``presenceStrs`` is present and none of the strings from ``absenceStrs`` are present in the HTML response
+- ``status_code`` - checks that status code of the response is 2XX
+- ``response_url`` - check if there is not redirect and the response is 2XX
+
+See the details of check mechanisms in the `checking.py <https://github.com/soxoj/maigret/blob/main/maigret/checking.py#L339>`_ file.
+
 Testing
 -------
 
-It is recommended use Python 3.7/3.8 for test due to some conflicts in 3.9.
+It is recommended use Python 3.10 for testing.
 
 Install test requirements:
 
@@ -20,20 +41,68 @@ Use the following commands to check Maigret:
 .. code-block:: console
 
   # run linter and typing checks
-  # order of checks%
+  # order of checks:
   # - critical syntax errors or undefined names
   # - flake checks
   # - mypy checks
   make lint
 
   # run testing with coverage html report
-  # current test coverage is 60%
-  make text
+  # current test coverage is 58%
+  make test
 
   # open html report
   open htmlcov/index.html
 
 
+How to fix false-positives
+-----------------------------------------------
+
+1. Determine the problematic site.
+
+If you already know which site has a false-positive and want to fix it specifically, go to the next step.
+
+Otherwise, simply run a search with a random username (e.g. `laiuhi3h4gi3u4hgt`) and check the results.
+Alternatively, you can use `the Telegram bot <https://t.me/osint_maigret_bot>`_.
+
+2. Open the account link in your browser and check:
+
+- If the site is completely gone, remove it from the list
+- If the site still works but looks different, update in data.json how we check it
+- If the site requires login to view profiles, disable checking it
+
+3. Find the site in the `data.json <https://github.com/soxoj/maigret/blob/main/maigret/resources/data.json>`_ file.
+
+If the ``checkType`` method is not ``message`` and you are going to fix check, update it:
+- put ``message`` in ``checkType``
+- put in ``absenceStrs`` a keyword that is present in the HTML response for an non-existing account
+- put in ``presenceStrs`` a keyword that is present in the HTML response for an existing account
+
+If you have trouble determining the right keywords, you can use automatic detection by passing the account URL with the ``--submit`` option:
+
+.. code-block:: console
+
+  maigret --submit https://my.mail.ru/bk/alex
+
+To disable checking, set ``disabled`` to ``true`` or simply run:
+
+.. code-block:: console
+
+  maigret --self-check --site My.Mail.ru@bk.ru
+
+To debug the check method using the response HTML, you can run:
+
+.. code-block:: console
+
+  maigret soxoj --site My.Mail.ru@bk.ru -d 2> response.txt
+
+There are few options for sites data.json helpful in various cases:
+
+- ``engine`` - a predefined check for the sites of certain type (e.g. forums), see the ``engines`` section in the JSON file
+- ``headers`` - a dictionary of additional headers to be sent to the site
+- ``requestHeadOnly`` - set to ``true`` if it's enough to make a HEAD request to the site
+- ``regexCheck`` - a regex to check if the username is valid, in case of frequent false-positives
+
 How to publish new version of Maigret
 -------------------------------------
 
@@ -98,4 +167,17 @@ PyPi package.
 - Press `+ Auto-generate release notes`
 - **Press "Publish release" button**
 
-8. That's all, now you can simply wait push to PyPi. You can monitor it in Action page: https://github.com/soxoj/maigret/actions/workflows/python-publish.yml
+8. That's all, now you can simply wait push to PyPi. You can monitor it in Action page: https://github.com/soxoj/maigret/actions/workflows/python-publish.yml
+
+Documentation updates
+--------------------
+
+Documentations is auto-generated and auto-deployed from the ``docs`` directory.
+
+To manually update documentation:
+
+1. Change something in the ``.rst`` files in the ``docs/source`` directory.
+2. Install ``pip install -r requirements.txt`` in the docs directory.
+3. Run ``make singlehtml`` in the terminal in the docs directory.
+4. Open ``build/singlehtml/index.html`` in your browser to see the result.
+5. If everything is ok, commit and push your changes to GitHub. 

docs/source/index.rst

@@ -3,11 +3,12 @@
 Welcome to the Maigret docs!
 ============================
 
-**Maigret** is an easy-to-use and powerful OSINT tool for collecting a dossier on a person by username only.
+**Maigret** is an easy-to-use and powerful OSINT tool for collecting a dossier on a person by a username (alias) only.
 
 This is achieved by checking for accounts on a huge number of sites and gathering all the available information from web pages.
 
-The project's main goal - give to OSINT researchers and pentesters a **universal tool** to get maximum information about a subject and integrate it with other tools in automatization pipelines.
+The project's main goal — give to OSINT researchers and pentesters a **universal tool** to get maximum information
+about a person of interest by a username and integrate it with other tools in automatization pipelines.
 
 You may be interested in:
 -------------------------

docs/source/roadmap.rst

@@ -3,6 +3,8 @@
 Roadmap
 =======
 
+**This roadmap is outdated and needs to be updated.**
+
 .. figure:: https://i.imgur.com/kk8cFdR.png   
    :target: https://i.imgur.com/kk8cFdR.png
    :align: center