mirror of
https://github.com/soxoj/maigret.git
synced 2026-05-06 14:08:59 +00:00
Sites re-check (#2423)
This commit is contained in:
Soxoj
@@ -115,6 +115,8 @@ Do **not** paste secrets, cookies, or full private JSON; short key names and str
|
||||
|
||||
### Phase C — Edits in [`data.json`](../maigret/resources/data.json)
|
||||
|
||||
**CRITICAL — surgical edits only.** `data.json` is a ~36 000-line file. **Never** rewrite it via `json.load()` + `json.dump()` — this reformats every line and produces a 70 000-line diff that is impossible to review. Instead, make **targeted text-level edits** (find the site's block, change only the specific lines). Use the `Edit` tool (or equivalent line-precise method), not a full JSON round-trip. The same rule applies to scripts: if a helper writes `data.json`, it must preserve the original formatting of untouched entries.
|
||||
|
||||
1. Update `url` / `urlMain` if needed (HTTPS, new profile path).
|
||||
2. Replace inappropriate `status_code` with `message` (or `response_url`), choosing:
|
||||
- **`absenceStrs`** — only what reliably appears on the “user does not exist” page;
|
||||
@@ -360,9 +362,25 @@ No need for fragile string matching when the API speaks HTTP correctly.
|
||||
|
||||
### 7.8 Engine templates can silently break across many sites
|
||||
|
||||
The **vBulletin** engine template has `absenceStrs` in five languages ("This user has not registered…", "Пользователь не зарегистрирован…", etc.). In a batch review of ~12 vBulletin forums (oneclickchicks, mirf, Pesiq, VKMOnline, forum.zone-game.info, etc.), **none** of the absence strings matched — the forums returned identical pages for both claimed and unclaimed usernames. Root cause: many of these forums require login to view member profiles, so they serve a generic page (no "user not registered" message at all) instead of an informative error.
|
||||
The **vBulletin** engine template has `absenceStrs` in six languages ("This user has not registered…", two Russian variants, Turkish, Ukrainian, Dutch). In a comprehensive audit of all 57 enabled vBulletin sites (2026-03-27), **26 were broken** (46%). The root causes were **not** template marker mismatch in most cases:
|
||||
|
||||
**Lesson:** When a whole engine class shows false positives, do not patch sites one by one — check whether the **engine template** itself still matches the actual error pages. A template written for one version/language pack may silently stop working after a forum upgrade or config change.
|
||||
| Category | Count | Examples |
|
||||
|----------|-------|---------|
|
||||
| Cloudflare challenge (403 `cf-mitigated`) | 7 | Mpgh, TheStudentRoom, SevenForums, alliedmods |
|
||||
| Dead/unreachable | 5 | Tanks, holodforum.ru, Microchip |
|
||||
| Server-side 403 (non-CF) | 5 | scaleforum.ru, forum-history.ru, Gorod.dp.ua |
|
||||
| Redirect/domain moved | 5 | Warface, Revelation, Stratege |
|
||||
| Login required to view profiles | 4 | goha, Animeforum, WiredNewYork |
|
||||
|
||||
Only the "login required" category relates to the template markers: when a forum requires authentication, the member.php page shows a generic response without the "user not registered" text. All 26 sites were disabled.
|
||||
|
||||
**Note on Russian translations:** Two distinct Russian vBulletin translations exist in the wild:
|
||||
- `"Этот пользователь ещё не зарегистрирован, поэтому его профиль недоступен."` (standard)
|
||||
- `"Пользователь не зарегистрирован и не имеет профиля для просмотра."` (goha.ru variant)
|
||||
|
||||
Both are now in the engine template.
|
||||
|
||||
**Lesson:** When a whole engine class shows high failure rates, categorize failures first — most are site-level infrastructure issues (CF, dead, auth), not template problems. Batch-disable broken sites rather than patching individually. Only investigate the template itself if the HTTP response is 200 but markers don't match.
|
||||
|
||||
### 7.9 Search-by-author URLs are architecturally unreliable
|
||||
|
||||
@@ -447,6 +465,74 @@ When a site's API returns a rate-limit response, the text may **not** match the
|
||||
|
||||
**Discord example (2026-03-24):** The POST API at `discord.com/api/v9/unique-username/username-attempt-unauthed` returns `{"taken":true}` / `{"taken":false}` normally, but under load returns varying rate-limit messages. Keeping only `{"taken":false}` in `absenceStrs` and all rate-limit variants in `errors` eliminates the transient false positives the Maigret bot was reporting.
|
||||
|
||||
### 7.16 Non-UTF-8 page encoding silently breaks string markers
|
||||
|
||||
**opennet.ru** serves pages in **KOI8-R** encoding. The `absenceStrs` value `"Имя участника не найдено"` is stored as UTF-8 bytes in `data.json`, but the HTTP response body contains the same text encoded as KOI8-R bytes. Since Maigret (and aiohttp) compares raw bytes by default, the substring is **never found** — the absence check silently fails, and empty `presenseStrs` (presence always true) produces a false CLAIMED.
|
||||
|
||||
**How to detect:** If `absenceStrs` contains non-ASCII text and the check fails despite the string visibly appearing on the page in a browser, inspect the `Content-Type` header or raw bytes for a non-UTF-8 `charset` (KOI8-R, Windows-1251, ISO-8859-*, etc.). Also check with `curl -s URL | iconv -f KOI8-R -t UTF-8` to confirm.
|
||||
|
||||
**Lesson:** Maigret has no built-in charset transcoding for marker comparison. If a site serves a non-UTF-8 charset and the relevant markers contain non-ASCII characters, string matching will fail. Options:
|
||||
- Find ASCII-only markers that work in any encoding (HTML tags, class names, English text).
|
||||
- Use a JSON API endpoint (APIs almost always return UTF-8).
|
||||
- If neither is available, `disabled: true`.
|
||||
|
||||
### 7.17 ARIA and HTML boilerplate attributes are dangerous `presenseStrs`
|
||||
|
||||
SlideShare had `"polite"` in `presenseStrs`, matching the standard `aria-live="polite"` attribute. This attribute appears on virtually any modern web page — including anti-bot challenge pages, error pages, and homepage redirects. When the real profile page is replaced by such a generic page, `absenceStrs` don't match (different content) but `presenseStrs` still fires → false CLAIMED.
|
||||
|
||||
**Common traps:** `polite`, `alert`, `status`, `navigation`, `assertive`, `banner`, `main`, `complementary`, `contentinfo` — all standard ARIA landmark/live-region values present on most pages.
|
||||
|
||||
**Lesson:** Never use single generic words that are part of HTML/ARIA boilerplate as `presenseStrs`. Profile markers should be **specific to the profile page structure**: unique CSS classes (e.g. `"profile-card"`), `<title>` fragments with the site name (e.g. `"- salon24.pl</title>"`), or JSON field names from API responses (e.g. `"displayName"`).
|
||||
|
||||
### 7.18 Anti-bot challenge pages can pass through `message` checks as false CLAIMED
|
||||
|
||||
When a site intermittently serves an anti-bot challenge page (e.g. SlideShare's "Client Challenge", Cloudflare "Just a moment..."), a specific failure mode occurs with `checkType: "message"`:
|
||||
|
||||
1. The challenge HTML replaces the real profile/error page.
|
||||
2. `absenceStrs` don't match (challenge page has different content than "user not found").
|
||||
3. If `presenseStrs` is empty (presence always true) **or** contains a broad marker that matches the challenge HTML → result is **CLAIMED**.
|
||||
|
||||
This is different from a simple "anti-bot → disable" situation because the challenge may be **intermittent** — the check works most of the time but produces sporadic false positives under load or for specific IPs.
|
||||
|
||||
**Fix:** Add the challenge page's distinctive text to `errors`:
|
||||
```json
|
||||
{
|
||||
"errors": {
|
||||
"Client Challenge": "Anti-bot challenge",
|
||||
"Just a moment": "Cloudflare challenge",
|
||||
"Checking your browser": "Anti-bot challenge"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `errors` mechanism produces **UNKNOWN** instead of CLAIMED, which is correct: "we got a challenge page, not a profile page, so we don't know."
|
||||
|
||||
**Lesson:** When fixing a site that is **intermittently** reported as false positive, check whether the failure happens only when anti-bot protection triggers. If so, adding challenge markers to `errors` is better than disabling the entire check.
|
||||
|
||||
### 7.19 Redirect-to-homepage as a "user not found" signal
|
||||
|
||||
Some sites (e.g. **Salon24.pl**) redirect non-existent user URLs to the **homepage** via HTTP 301/302, while existing users get a 200 with profile content. Since Maigret follows redirects by default (`allow_redirects=True` for `message`/`status_code` checks), it sees the **final** page — the homepage.
|
||||
|
||||
This creates a usable signal for `checkType: "message"`:
|
||||
- **`presenseStrs`** with a fragment unique to profile pages (e.g. `"- salon24.pl</title>"` which appears in `"test 1 - salon24.pl</title>"` on profiles but not on the generic homepage title).
|
||||
- No `absenceStrs` needed — the homepage simply doesn't contain the profile-specific marker.
|
||||
|
||||
**Lesson:** When a site returns the same HTTP 200 for both users (after redirect-follow), compare the **final page content** for both. If unclaimed lands on the homepage, use a profile-specific `presenseStrs` marker rather than trying to find an absence string on the homepage.
|
||||
|
||||
### 7.20 Non-standard HTTP status codes from anti-bot systems
|
||||
|
||||
Anti-bot systems don't always use standard 403/429 codes. Observed examples:
|
||||
- **HTTP 468** (forum.exkavator.ru) — custom Tengine anti-bot status.
|
||||
- **HTTP 520–530** — Cloudflare-specific error codes (520 = unknown error, 521 = web server down, 522 = connection timed out, 523 = origin unreachable, 524 = timeout, 525 = SSL handshake failed, 526 = invalid SSL, 530 = with 1xxx error).
|
||||
|
||||
**Lesson:** When diagnosing a site that returns connection errors or unexpected statuses in Maigret, check with `curl -sIL` first. If the status code is non-standard (not 2xx/3xx/4xx/5xx from the origin), it's likely an intermediary (CDN, WAF, anti-bot) and the site should be `disabled: true`.
|
||||
|
||||
### 7.21 `site_check.py --diagnose` does not test POST APIs
|
||||
|
||||
The `utils/site_check.py --diagnose` tool performs raw aiohttp GET requests to compare claimed/unclaimed responses. For sites that use `requestMethod: "POST"` (e.g. Discord, Holopin), the diagnose tool will show the site as broken because GET to a POST endpoint returns different content (often the site's homepage or an error page).
|
||||
|
||||
**Workaround:** For POST-based checks, verify manually with `curl -X POST` or use `maigret --self-check --site "SiteName"` which respects the full configuration including request method and payload.
|
||||
|
||||
### 7.7 The playbook classification works
|
||||
|
||||
The decision tree from the documentation accurately describes real-world cases:
|
||||
|
||||
@@ -21,18 +21,26 @@ Short checklist for edits to [`maigret/resources/data.json`](../maigret/resource
|
||||
| Symptom | Typical cause | Action |
|
||||
|--------|-----------------|--------|
|
||||
| HTTP 200 for “user does not exist” | Soft 404 | Move from `status_code` to `message` or `response_url`; add `absenceStrs` / narrow `presenseStrs` |
|
||||
| Generic words match (`name`, `email`) | `presenseStrs` too broad | Remove generic markers; add profile-specific ones |
|
||||
| Generic words match (`name`, `email`) | `presenseStrs` too broad | Remove generic markers; add profile-specific ones. **Avoid** ARIA/boilerplate words (`polite`, `alert`, `navigation`, etc.) — see 7.17 in guide |
|
||||
| Same HTML without JS | SPA / skeleton shell | Compare **final URL and HTTP redirects** (Maigret already follows redirects by default). If the browser shows extra routes (`/posts`, `/not-found`) only **after JS**, they will **not** appear to Maigret — try a **public JSON/API** endpoint for the same site if one exists. See **Redirects and final URL** and **Picsart** in [`site-checks-guide.md`](site-checks-guide.md). |
|
||||
| Unclaimed redirects to homepage | Site returns 301/302 to main page | Use `presenseStrs` with a profile-specific marker (e.g. title fragment unique to profile pages). See 7.19 in guide |
|
||||
| 403 / “Log in” / guest-only | Auth or anti-bot required | `disabled: true` |
|
||||
| reCAPTCHA / “Checking your browser” | Bot protection | Try a reasonable `User-Agent` in `headers`; else `errors` + UNKNOWN or `disabled` |
|
||||
| reCAPTCHA / “Checking your browser” / “Client Challenge” | Bot protection | Add challenge text to `errors` (→ UNKNOWN). Try a reasonable `User-Agent` in `headers`. If intermittent, `errors` is better than `disabled`. See 7.18 in guide |
|
||||
| Non-standard HTTP code (468, 520–530) | CDN/WAF anti-bot | `disabled: true`. Check with `curl -sIL` to confirm the code comes from an intermediary. See 7.20 in guide |
|
||||
| Non-ASCII `absenceStrs` not matching despite visible text | Page encoding ≠ UTF-8 | Check `Content-Type` for charset (KOI8-R, Windows-1251, etc.). Use ASCII-only markers, a JSON API, or `disabled: true`. See 7.16 in guide |
|
||||
| Domain does not resolve / persistent timeout | Dead service | Remove entry **only** after confirming the domain is dead |
|
||||
|
||||
## 3. Data edits
|
||||
|
||||
**CRITICAL — surgical edits only.** Never rewrite `data.json` via `json.load()` + `json.dump()` — this reformats the entire ~36 000-line file and produces an unreviewable diff. Make targeted, line-level edits to only the fields you are changing. See Phase C in [`site-checks-guide.md`](site-checks-guide.md).
|
||||
|
||||
1. Update `url` / `urlMain` if needed (HTTPS redirects). Use optional **`urlProbe`** when the HTTP check should hit a different URL than the profile link shown in reports (API vs web UI).
|
||||
2. For `message`: **always** tune string pairs so `absenceStrs` fire on “no user” pages and `presenseStrs` fire on real profiles without false absence hits.
|
||||
- **Never** use ARIA/boilerplate words as `presenseStrs` (`polite`, `alert`, `navigation`, `status`, `main`, etc.).
|
||||
- If markers contain **non-ASCII text**, verify the page charset is UTF-8. Non-UTF-8 pages (KOI8-R, Windows-1251) will silently fail byte comparison — prefer ASCII-only markers or a JSON API.
|
||||
3. Engine (`engine`, e.g. XenForo): override only differing fields in the site entry so other sites are not broken.
|
||||
4. Keep `status_code` only if the response **reliably** differs by status code without soft 404.
|
||||
5. Add **anti-bot challenge text** to `errors` (not `absenceStrs`) when the site intermittently serves challenge pages. Common patterns: `”Client Challenge”`, `”Just a moment”`, `”Checking your browser”`, `”Attention Required”`. This produces UNKNOWN instead of false CLAIMED.
|
||||
|
||||
## 4. Verify
|
||||
|
||||
@@ -82,6 +90,12 @@ Practical observations from fixing top-ranked sites. Full details: section **7**
|
||||
| **URL-encode braces for template safety** | GraphQL `{...}` conflicts with Maigret's `{username}`. Use `%7B`/`%7D` for literal braces in `urlProbe` — `.format()` ignores percent-encoded chars. |
|
||||
| **Anti-bot bypass via simple UA** | "Anubis" anti-bot PoW screens (like on Weblate) intercept modern browser UAs via HTTP 307. Hardcoding `"headers": {"User-Agent": "python-requests/2.25.1"}` circumvents the scraper filter and restores default detection logic. |
|
||||
| **Rate-limit → `errors`, not `absenceStrs`** | Rate-limit wording varies across API versions. If the phrasing doesn't match `absenceStrs` and `presenseStrs` is empty, the result is a false CLAIMED. Put all "can't answer right now" strings (rate limit, CAPTCHA, maintenance) in `errors` so the result is UNKNOWN. |
|
||||
| **Non-UTF-8 encoding breaks markers** | opennet.ru serves KOI8-R; UTF-8 `absenceStrs` never match raw bytes. Use ASCII-only markers, a JSON API, or `disabled: true`. |
|
||||
| **ARIA attrs are presenseStrs traps** | `"polite"`, `"alert"`, `"navigation"` match `aria-live`/ARIA landmarks on any page including anti-bot challenges. Use profile-specific markers instead. |
|
||||
| **Anti-bot challenge + broad markers = false CLAIMED** | Challenge pages bypass `absenceStrs` but match broad `presenseStrs`. Add challenge text (e.g. `"Client Challenge"`) to `errors` → UNKNOWN. Better than disabling for intermittent issues. |
|
||||
| **Redirect-to-homepage as signal** | Salon24.pl 301-redirects unclaimed users to homepage. Use `presenseStrs` with a profile-only marker (e.g. `"- salon24.pl</title>"`). |
|
||||
| **Non-standard anti-bot HTTP codes** | HTTP 468 (Tengine), 520–530 (Cloudflare) — not standard 403/429. Check with `curl -sIL`; if code is from intermediary → `disabled: true`. |
|
||||
| **`--diagnose` doesn't test POST** | `site_check.py --diagnose` uses GET only. For POST APIs (Discord, Holopin), verify with `curl -X POST` or `maigret --self-check`. |
|
||||
|
||||
## 8. Documentation maintenance
|
||||
|
||||
|
||||
+63
-20
@@ -1052,7 +1052,6 @@
|
||||
"user-name",
|
||||
"pageInfo",
|
||||
"listitem",
|
||||
"polite",
|
||||
"strippedTitle"
|
||||
],
|
||||
"absenceStrs": [
|
||||
@@ -1060,7 +1059,10 @@
|
||||
"username-available",
|
||||
"robots",
|
||||
"noindex,nofollow"
|
||||
]
|
||||
],
|
||||
"errors": {
|
||||
"Client Challenge": "Anti-bot challenge"
|
||||
}
|
||||
},
|
||||
"Behance": {
|
||||
"tags": [
|
||||
@@ -2386,6 +2388,7 @@
|
||||
},
|
||||
"Revelation": {
|
||||
"urlSubpath": "/forums",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"gaming",
|
||||
@@ -2398,6 +2401,7 @@
|
||||
},
|
||||
"Tanks": {
|
||||
"urlSubpath": "/forum",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"gaming",
|
||||
@@ -2410,6 +2414,7 @@
|
||||
},
|
||||
"Warface": {
|
||||
"urlSubpath": "/forums",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"ru"
|
||||
@@ -3580,6 +3585,7 @@
|
||||
]
|
||||
},
|
||||
"Gorod.dp.ua": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"de",
|
||||
"forum",
|
||||
@@ -11220,6 +11226,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"TheStudentRoom": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"gb"
|
||||
@@ -11886,6 +11893,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"alliedmods": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"gb",
|
||||
@@ -12407,6 +12415,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"SevenForums": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"gb",
|
||||
@@ -12788,6 +12797,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"opennet": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"fr",
|
||||
"ru"
|
||||
@@ -13629,6 +13639,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"BlackHatProTools": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum"
|
||||
],
|
||||
@@ -13947,6 +13958,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"goha": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"ru"
|
||||
@@ -17163,6 +17175,7 @@
|
||||
},
|
||||
"AllTheLyrics": {
|
||||
"urlSubpath": "/forum",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"music"
|
||||
@@ -17326,6 +17339,7 @@
|
||||
},
|
||||
"Mpgh": {
|
||||
"urlSubpath": "/forum",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"jp",
|
||||
@@ -17387,6 +17401,7 @@
|
||||
},
|
||||
"Movie-list": {
|
||||
"urlSubpath": "/forum",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"ca",
|
||||
"forum",
|
||||
@@ -17806,6 +17821,7 @@
|
||||
},
|
||||
"Coolminiornot": {
|
||||
"urlSubpath": "/forums",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"sg",
|
||||
@@ -17885,6 +17901,7 @@
|
||||
},
|
||||
"Stratege": {
|
||||
"urlSubpath": "/forums",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"gaming",
|
||||
@@ -18490,6 +18507,7 @@
|
||||
},
|
||||
"WiredNewYork": {
|
||||
"urlSubpath": "/forum",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"in",
|
||||
@@ -18947,6 +18965,7 @@
|
||||
"alexaRank": 227287
|
||||
},
|
||||
"Animeforum": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"pk",
|
||||
@@ -19020,6 +19039,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"EduGeek": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"education",
|
||||
"us"
|
||||
@@ -20175,10 +20195,7 @@
|
||||
"ir"
|
||||
],
|
||||
"regexCheck": "^[a-zA-Z0-9]+(-[a-zA-Z0-9]+)*$",
|
||||
"checkType": "message",
|
||||
"absenceStrs": [
|
||||
"Profile Not Found"
|
||||
],
|
||||
"checkType": "status_code",
|
||||
"alexaRank": 328753,
|
||||
"urlMain": "https://www.house-mixes.com/",
|
||||
"url": "https://www.house-mixes.com/profile/{username}",
|
||||
@@ -20238,6 +20255,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"forum.exkavator.ru": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"ru"
|
||||
@@ -21286,6 +21304,7 @@
|
||||
"disabled": true
|
||||
},
|
||||
"Goldroyal": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"bd",
|
||||
"by",
|
||||
@@ -21811,6 +21830,7 @@
|
||||
"checkType": "message"
|
||||
},
|
||||
"Msofficeforums": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"ir",
|
||||
@@ -22764,6 +22784,7 @@
|
||||
]
|
||||
},
|
||||
"scaleforum.ru": {
|
||||
"disabled": true,
|
||||
"urlMain": "http://www.scaleforum.ru",
|
||||
"engine": "vBulletin",
|
||||
"usernameClaimed": "alex",
|
||||
@@ -22798,6 +22819,7 @@
|
||||
},
|
||||
"Indog": {
|
||||
"urlSubpath": "/forum",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"ru"
|
||||
@@ -23208,6 +23230,7 @@
|
||||
"alexaRank": 750321
|
||||
},
|
||||
"holodforum.ru": {
|
||||
"disabled": true,
|
||||
"urlMain": "https://holodforum.ru",
|
||||
"engine": "vBulletin",
|
||||
"usernameClaimed": "alex",
|
||||
@@ -25506,14 +25529,9 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"Dojoverse": {
|
||||
"checkType": "message",
|
||||
"absenceStrs": [
|
||||
"Looks like you got lost!."
|
||||
],
|
||||
"presenseStrs": [
|
||||
"Joined"
|
||||
],
|
||||
"checkType": "status_code",
|
||||
"url": "https://dojoverse.com/members/{username}/",
|
||||
"urlMain": "https://dojoverse.com",
|
||||
"usernameClaimed": "uehkon89",
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
@@ -26148,6 +26166,7 @@
|
||||
"disabled": true
|
||||
},
|
||||
"Flbord": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"ru",
|
||||
"ua"
|
||||
@@ -26394,6 +26413,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"GDProfiles": {
|
||||
"disabled": true,
|
||||
"checkType": "status_code",
|
||||
"urlMain": "https://gdprofiles.com/",
|
||||
"url": "https://gdprofiles.com/{username}",
|
||||
@@ -27090,8 +27110,13 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"Lesswrong": {
|
||||
"checkType": "status_code",
|
||||
"checkType": "message",
|
||||
"presenseStrs": [
|
||||
"displayName"
|
||||
],
|
||||
"url": "https://www.lesswrong.com/users/@{username}",
|
||||
"urlMain": "https://www.lesswrong.com",
|
||||
"urlProbe": "https://www.lesswrong.com/graphql?query=%7B%20user(input%3A%20%7Bselector%3A%20%7Bslug%3A%20%22{username}%22%7D%7D)%20%7B%20result%20%7B%20displayName%20%7D%20%7D%20%7D",
|
||||
"usernameClaimed": "blue",
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
@@ -27538,6 +27563,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"Microchip": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"ru"
|
||||
@@ -27593,6 +27619,7 @@
|
||||
},
|
||||
"Mobile-files": {
|
||||
"urlSubpath": "/forum",
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"ru",
|
||||
@@ -28596,8 +28623,12 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"Salon24.pl": {
|
||||
"checkType": "status_code",
|
||||
"checkType": "message",
|
||||
"presenseStrs": [
|
||||
"- salon24.pl</title>"
|
||||
],
|
||||
"url": "https://www.salon24.pl/u/{username}/",
|
||||
"urlMain": "https://www.salon24.pl",
|
||||
"usernameClaimed": "test",
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
@@ -29088,6 +29119,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"Syktforum": {
|
||||
"disabled": true,
|
||||
"tags": [
|
||||
"forum",
|
||||
"ru"
|
||||
@@ -29319,6 +29351,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"Thephysicsforum": {
|
||||
"disabled": true,
|
||||
"checkType": "message",
|
||||
"absenceStrs": [
|
||||
"This user has not registered and therefore does not have a profile to view."
|
||||
@@ -32925,6 +32958,7 @@
|
||||
]
|
||||
},
|
||||
"rub.altai.su": {
|
||||
"disabled": true,
|
||||
"urlMain": "http://rub.altai.su",
|
||||
"engine": "vBulletin",
|
||||
"usernameClaimed": "alex",
|
||||
@@ -32988,6 +33022,7 @@
|
||||
"disabled": true
|
||||
},
|
||||
"porschec.ru": {
|
||||
"disabled": true,
|
||||
"urlMain": "http://porschec.ru",
|
||||
"engine": "vBulletin",
|
||||
"usernameClaimed": "alex",
|
||||
@@ -33295,6 +33330,7 @@
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
"forum-history.ru": {
|
||||
"disabled": true,
|
||||
"urlMain": "http://forum-history.ru",
|
||||
"engine": "vBulletin",
|
||||
"usernameClaimed": "red",
|
||||
@@ -33940,8 +33976,10 @@
|
||||
"usernameClaimed": "adam",
|
||||
"usernameUnclaimed": "noonewouldeverusethis7",
|
||||
"url": "https://en.brickimedia.org/wiki/User:{username}",
|
||||
"urlMain": "https://en.brickimedia.org",
|
||||
"urlProbe": "https://en.brickimedia.org/w/api.php?action=query&list=users&ususers={username}&format=json",
|
||||
"absenceStrs": [
|
||||
"noarticletext"
|
||||
"\"missing\""
|
||||
]
|
||||
},
|
||||
"en.illogicopedia.org": {
|
||||
@@ -35868,6 +35906,7 @@
|
||||
]
|
||||
},
|
||||
"Codolio": {
|
||||
"disabled": true,
|
||||
"url": "https://codolio.com/profile/{username}",
|
||||
"urlMain": "https://codolio.com/",
|
||||
"checkType": "message",
|
||||
@@ -35924,8 +35963,8 @@
|
||||
"urlMain": "https://holopin.io",
|
||||
"urlProbe": "https://www.holopin.io/api/auth/username",
|
||||
"checkType": "message",
|
||||
"absenceStrs": [
|
||||
"true"
|
||||
"presenseStrs": [
|
||||
"\"available\":false"
|
||||
],
|
||||
"usernameClaimed": "red",
|
||||
"usernameUnclaimed": "noonewouldeverusethis7",
|
||||
@@ -35983,7 +36022,10 @@
|
||||
"Shelf": {
|
||||
"url": "https://www.shelf.im/{username}",
|
||||
"urlMain": "https://www.shelf.im/",
|
||||
"checkType": "status_code",
|
||||
"checkType": "message",
|
||||
"absenceStrs": [
|
||||
"User Not Found"
|
||||
],
|
||||
"usernameClaimed": "blue",
|
||||
"usernameUnclaimed": "noonewouldeverusethis7"
|
||||
},
|
||||
@@ -36167,7 +36209,8 @@
|
||||
"Bu \u00dcye kay\u0131tl\u0131 \u00dcyemiz de\u011fildir. Bu sebebten dolay\u0131 \u00dcyeye ait Profil g\u00f6sterilemiyor.",
|
||||
"This user has not registered and therefore does not have a profile to view.",
|
||||
"\u041a\u043e\u0440\u0438\u0441\u0442\u0443\u0432\u0430\u0447 \u043d\u0435 \u0437\u0430\u0440\u0435\u0454\u0441\u0442\u0440\u043e\u0432\u0430\u043d\u0438\u0439 \u0456 \u043d\u0435 \u043c\u0430\u0454 \u043f\u0440\u043e\u0444\u0456\u043b\u044e, \u044f\u043a\u0438\u0439 \u043c\u043e\u0436\u043d\u0430 \u043f\u0435\u0440\u0435\u0433\u043b\u044f\u043d\u0443\u0442\u0438.",
|
||||
"Deze gebruiker is niet geregistreerd, zodat je zijn of haar profiel niet kunt bekijken."
|
||||
"Deze gebruiker is niet geregistreerd, zodat je zijn of haar profiel niet kunt bekijken.",
|
||||
"\u042d\u0442\u043e\u0442 \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044c \u0435\u0449\u0451 \u043d\u0435 \u0437\u0430\u0440\u0435\u0433\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d, \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u0435\u0433\u043e \u043f\u0440\u043e\u0444\u0438\u043b\u044c \u043d\u0435\u0434\u043e\u0441\u0442\u0443\u043f\u0435\u043d."
|
||||
],
|
||||
"checkType": "message",
|
||||
"errors": {
|
||||
|
||||
@@ -417,7 +417,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [PushSquare (http://www.pushsquare.com)](http://www.pushsquare.com)*: top 100K, gaming, news, us*
|
||||
1.  [Coderwall (https://coderwall.com/)](https://coderwall.com/)*: top 100K, coding, in*
|
||||
1.  [Otzovik (https://otzovik.com/)](https://otzovik.com/)*: top 100K, ru*
|
||||
1.  [TheStudentRoom (https://www.thestudentroom.co.uk)](https://www.thestudentroom.co.uk)*: top 100K, forum, gb*
|
||||
1.  [TheStudentRoom (https://www.thestudentroom.co.uk)](https://www.thestudentroom.co.uk)*: top 100K, forum, gb*, search is disabled
|
||||
1.  [lomography (https://www.lomography.com)](https://www.lomography.com)*: top 100K, photo*
|
||||
1.  [Observable (https://observablehq.com)](https://observablehq.com)*: top 100K, sharing*
|
||||
1.  [Codementor (https://www.codementor.io/)](https://www.codementor.io/)*: top 100K, coding, in*
|
||||
@@ -485,7 +485,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [TryHackMe (https://tryhackme.com/)](https://tryhackme.com/)*: top 100K, hacking, us*
|
||||
1.  [nhattao.com (https://www.nhattao.com)](https://www.nhattao.com)*: top 100K, forum, shopping, vn*
|
||||
1.  [Voices (https://www.voices.com/)](https://www.voices.com/)*: top 100K, us*
|
||||
1.  [SevenForums (https://www.sevenforums.com)](https://www.sevenforums.com)*: top 100K, forum, gb, us*
|
||||
1.  [SevenForums (https://www.sevenforums.com)](https://www.sevenforums.com)*: top 100K, forum, gb, us*, search is disabled
|
||||
1.  [club.cnews.ru (https://club.cnews.ru/)](https://club.cnews.ru/)*: top 100K, blog, ru*, search is disabled
|
||||
1.  [Dcinside (https://www.dcinside.com/)](https://www.dcinside.com/)*: top 100K*
|
||||
1.  [Repl.it (https://repl.it/)](https://repl.it/)*: top 100K, coding, us*
|
||||
@@ -509,7 +509,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [3dtoday (https://3dtoday.ru/)](https://3dtoday.ru/)*: top 100K, ru*
|
||||
1.  [Influenster (https://www.influenster.com/)](https://www.influenster.com/)*: top 100K, us*, search is disabled
|
||||
1.  [gloria.tv (https://gloria.tv)](https://gloria.tv)*: top 100K, ar, mx, pl, sk, us*
|
||||
1.  [opennet (https://www.opennet.ru/)](https://www.opennet.ru/)*: top 100K, fr, ru*
|
||||
1.  [opennet (https://www.opennet.ru/)](https://www.opennet.ru/)*: top 100K, fr, ru*, search is disabled
|
||||
1.  [Dealabs (https://www.dealabs.com/)](https://www.dealabs.com/)*: top 100K, de*
|
||||
1.  [Lobsters (https://lobste.rs/)](https://lobste.rs/)*: top 100K, in, us, vn*
|
||||
1.  [Tripline (https://www.tripline.net)](https://www.tripline.net)*: top 100K, in*
|
||||
@@ -554,7 +554,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [polarsteps (https://polarsteps.com/)](https://polarsteps.com/)*: top 100K, in, us*
|
||||
1.  [fl (https://www.fl.ru/)](https://www.fl.ru/)*: top 100K, ru*
|
||||
1.  [YouPic (https://youpic.com/)](https://youpic.com/)*: top 100K, photo*
|
||||
1.  [BlackHatProTools (https://www.blackhatprotools.info)](https://www.blackhatprotools.info)*: top 100K, forum*
|
||||
1.  [BlackHatProTools (https://www.blackhatprotools.info)](https://www.blackhatprotools.info)*: top 100K, forum*, search is disabled
|
||||
1.  [EyeEm (https://www.eyeem.com/)](https://www.eyeem.com/)*: top 100K, in, it, photo, sd*
|
||||
1.  [Virgool (https://virgool.io/)](https://virgool.io/)*: top 100K, blog, ir*, search is disabled
|
||||
1.  [Chatujme.cz (https://chatujme.cz/)](https://chatujme.cz/)*: top 100K, cz, dating*
|
||||
@@ -763,11 +763,11 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [Nothing Community (https://nothing.community/)](https://nothing.community/)*: top 10M, forum*
|
||||
1.  [Old-games (https://www.old-games.ru)](https://www.old-games.ru)*: top 10M, pt, ru*
|
||||
1.  [getmyuni (https://getmyuni.com/)](https://getmyuni.com/)*: top 10M, in*, search is disabled
|
||||
1.  [Mpgh (https://www.mpgh.net/)](https://www.mpgh.net/)*: top 10M, forum, jp, us*
|
||||
1.  [Mpgh (https://www.mpgh.net/)](https://www.mpgh.net/)*: top 10M, forum, jp, us*, search is disabled
|
||||
1.  [RoyalCams (https://royalcams.com)](https://royalcams.com)*: top 10M, gr, in, ng, ru, us, webcam*
|
||||
1.  [Au (https://au.ru)](https://au.ru)*: top 10M, freelance, ru, shopping*
|
||||
1.  [bitpapa.com (https://bitpapa.com)](https://bitpapa.com)*: top 10M, crypto*, search is disabled
|
||||
1.  [Movie-list (https://www.movie-list.com)](https://www.movie-list.com)*: top 10M, ca, forum, in, pk*
|
||||
1.  [Movie-list (https://www.movie-list.com)](https://www.movie-list.com)*: top 10M, ca, forum, in, pk*, search is disabled
|
||||
1.  [Football (https://www.rusfootball.info/)](https://www.rusfootball.info/)*: top 10M, ru*
|
||||
1.  [www.freelancejob.ru (https://www.freelancejob.ru)](https://www.freelancejob.ru)*: top 10M, ru*
|
||||
1.  [1001tracklists (https://www.1001tracklists.com)](https://www.1001tracklists.com)*: top 10M, us*
|
||||
@@ -790,9 +790,9 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [The AnswerBank (https://www.theanswerbank.co.uk)](https://www.theanswerbank.co.uk)*: top 10M, gb, q&a*
|
||||
1.  [Radiokot (https://www.radiokot.ru)](https://www.radiokot.ru)*: top 10M, ru*
|
||||
1.  [Modx_pro (https://modx.pro)](https://modx.pro)*: top 10M, ru, uz*
|
||||
1.  [Coolminiornot (http://www.coolminiornot.com)](http://www.coolminiornot.com)*: top 10M, forum, sg, us*
|
||||
1.  [Coolminiornot (http://www.coolminiornot.com)](http://www.coolminiornot.com)*: top 10M, forum, sg, us*, search is disabled
|
||||
1.  [Gramho (https://gramho.com/)](https://gramho.com/)*: top 10M, photo*, search is disabled
|
||||
1.  [Stratege (https://www.stratege.ru)](https://www.stratege.ru)*: top 10M, forum, gaming, news, ru*
|
||||
1.  [Stratege (https://www.stratege.ru)](https://www.stratege.ru)*: top 10M, forum, gaming, news, ru*, search is disabled
|
||||
1.  [Thebigboss (http://thebigboss.org)](http://thebigboss.org)*: top 10M, tr*
|
||||
1.  [Mylespaul (https://www.mylespaul.com)](https://www.mylespaul.com)*: top 10M, cl, us*
|
||||
1.  [Bdoutdoors (https://www.bdoutdoors.com)](https://www.bdoutdoors.com)*: top 10M, us*
|
||||
@@ -827,7 +827,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [oshwlab.com (https://oshwlab.com)](https://oshwlab.com)*: top 10M*
|
||||
1.  [sevenstring.org (https://sevenstring.org)](https://sevenstring.org)*: top 10M, forum*
|
||||
1.  [Snbforums (https://www.snbforums.com)](https://www.snbforums.com)*: top 10M, forum, us*
|
||||
1.  [WiredNewYork (http://wirednewyork.com/)](http://wirednewyork.com/)*: top 10M, forum, in, pk, us*
|
||||
1.  [WiredNewYork (http://wirednewyork.com/)](http://wirednewyork.com/)*: top 10M, forum, in, pk, us*, search is disabled
|
||||
1.  [pr0gramm (https://pr0gramm.com/)](https://pr0gramm.com/)*: top 10M, de*
|
||||
1.  [Stereo (https://stereo.ru/)](https://stereo.ru/)*: top 10M, nl, ru*
|
||||
1.  [www.opendiary.com (https://www.opendiary.com)](https://www.opendiary.com)*: top 10M, us*
|
||||
@@ -853,11 +853,11 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [railforums.co.uk (https://www.railforums.co.uk)](https://www.railforums.co.uk)*: top 10M, forum, jp*
|
||||
1.  [GotovimDoma (https://gotovim-doma.ru)](https://gotovim-doma.ru)*: top 10M, ru*
|
||||
1.  [bliphoto (https://www.blipfoto.com)](https://www.blipfoto.com)*: top 10M, photo*
|
||||
1.  [Animeforum (https://www.animeforum.com)](https://www.animeforum.com)*: top 10M, forum, pk, us, vn*
|
||||
1.  [Animeforum (https://www.animeforum.com)](https://www.animeforum.com)*: top 10M, forum, pk, us, vn*, search is disabled
|
||||
1.  [Hpc (https://hpc.ru)](https://hpc.ru)*: top 10M, in, ru*
|
||||
1.  [Pentester Lab (https://pentesterlab.com/)](https://pentesterlab.com/)*: top 10M*
|
||||
1.  [Ruby-forum (https://www.ruby-forum.com)](https://www.ruby-forum.com)*: top 10M, forum, us*
|
||||
1.  [EduGeek (https://www.edugeek.net)](https://www.edugeek.net)*: top 10M, education, us*
|
||||
1.  [EduGeek (https://www.edugeek.net)](https://www.edugeek.net)*: top 10M, education, us*, search is disabled
|
||||
1.  [justmj.ru (http://justmj.ru)](http://justmj.ru)*: top 10M*
|
||||
1.  [Empflix (https://www.empflix.com)](https://www.empflix.com)*: top 10M, de, fr, porn*
|
||||
1.  [Hackenproof (https://hackenproof.com/arbin)](https://hackenproof.com/arbin)*: top 10M, in, ua*
|
||||
@@ -997,7 +997,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [rezzoclub.ru (http://rezzoclub.ru)](http://rezzoclub.ru)*: top 10M*
|
||||
1.  [rybnoe.net (http://rybnoe.net)](http://rybnoe.net)*: top 10M, ru, ua*
|
||||
1.  [Pogovorim (https://pogovorim.by)](https://pogovorim.by)*: top 10M, by, ru*, search is disabled
|
||||
1.  [Goldroyal (http://goldroyal.net)](http://goldroyal.net)*: top 10M, bd, by, forum, ru, ua, ve*
|
||||
1.  [Goldroyal (http://goldroyal.net)](http://goldroyal.net)*: top 10M, bd, by, forum, ru, ua, ve*, search is disabled
|
||||
1.  [Warframe Market (https://warframe.market/)](https://warframe.market/)*: top 10M, gaming*
|
||||
1.  [Thelion (http://www.thelion.com)](http://www.thelion.com)*: top 10M, us*
|
||||
1.  [FCRubin (https://www.fcrubin.ru)](https://www.fcrubin.ru)*: top 10M, forum, ru*
|
||||
@@ -1029,7 +1029,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [Rusarmy (http://www.rusarmy.com)](http://www.rusarmy.com)*: top 10M, ru*
|
||||
1.  [ForumVancouver (http://www.forumvancouver.com)](http://www.forumvancouver.com)*: top 10M, ca, forum*
|
||||
1.  [directx10.org (http://directx10.org)](http://directx10.org)*: top 10M*
|
||||
1.  [Msofficeforums (https://www.msofficeforums.com)](https://www.msofficeforums.com)*: top 10M, forum, ir, us*
|
||||
1.  [Msofficeforums (https://www.msofficeforums.com)](https://www.msofficeforums.com)*: top 10M, forum, ir, us*, search is disabled
|
||||
1.  [Astrogalaxy (https://astrogalaxy.ru)](https://astrogalaxy.ru)*: top 10M, ru*, search is disabled
|
||||
1.  [stripchat.global (https://stripchat.global)](https://stripchat.global)*: top 10M, webcam*, search is disabled
|
||||
1.  [tigerfan.com (http://www.tigerfan.com/)](http://www.tigerfan.com/)*: top 10M, forum, sport*
|
||||
@@ -1092,10 +1092,10 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [VegasCreativeSoftware (https://www.vegascreativesoftware.info)](https://www.vegascreativesoftware.info)*: top 10M, us*, search is disabled
|
||||
1.  [PeopleAndCountries (http://peopleandcountries.com)](http://peopleandcountries.com)*: top 10M, forum, ru*, search is disabled
|
||||
1.  [Psyera (https://psyera.ru)](https://psyera.ru)*: top 10M, ru*
|
||||
1.  [scaleforum.ru (http://www.scaleforum.ru)](http://www.scaleforum.ru)*: top 10M*
|
||||
1.  [scaleforum.ru (http://www.scaleforum.ru)](http://www.scaleforum.ru)*: top 10M*, search is disabled
|
||||
1.  [Antiwomen (https://antiwomen.ru)](https://antiwomen.ru)*: top 10M, forum, ru*
|
||||
1.  [cowboyszone.com (https://cowboyszone.com)](https://cowboyszone.com)*: top 10M, forum, us*
|
||||
1.  [Indog (http://www.indog.ru/)](http://www.indog.ru/)*: top 10M, forum, ru*
|
||||
1.  [Indog (http://www.indog.ru/)](http://www.indog.ru/)*: top 10M, forum, ru*, search is disabled
|
||||
1.  [nokia-love.ru (http://nokia-love.ru)](http://nokia-love.ru)*: top 10M, ru*
|
||||
1.  [Totseans (http://www.totseans.com/bbs/profile/Vizier)](http://www.totseans.com/bbs/profile/Vizier)*: top 10M, forum*, search is disabled
|
||||
1.  [W3challs (https://w3challs.com/)](https://w3challs.com/)*: top 10M, tn, us*
|
||||
@@ -1123,7 +1123,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [Carmasters (https://carmasters.org)](https://carmasters.org)*: top 10M, fi, ru*
|
||||
1.  [Writercenter (https://writercenter.ru)](https://writercenter.ru)*: top 10M, ru, ua*
|
||||
1.  [Status Cafe (https://status.cafe/)](https://status.cafe/)*: top 10M*
|
||||
1.  [holodforum.ru (https://holodforum.ru)](https://holodforum.ru)*: top 10M, forum*
|
||||
1.  [holodforum.ru (https://holodforum.ru)](https://holodforum.ru)*: top 10M, forum*, search is disabled
|
||||
1.  [sputnikkey.ru (http://sputnikkey.ru)](http://sputnikkey.ru)*: top 10M*
|
||||
1.  [Erogen.club (https://erogen.club)](https://erogen.club)*: top 10M, forum, ru, ua*
|
||||
1.  [sanatatur.ru (http://sanatatur.ru/forum)](http://sanatatur.ru/forum)*: top 10M, forum, ru*
|
||||
@@ -1248,9 +1248,9 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [My.Mail.ru@yandex.ru (https://my.mail.ru/)](https://my.mail.ru/)*: top 100M, ru*
|
||||
1.  [PerfectWorld (https://pw.mail.ru)](https://pw.mail.ru)*: top 100M, gaming, ru*
|
||||
1.  [PerfectWorldForum (https://pw.mail.ru/)](https://pw.mail.ru/)*: top 100M, forum, gaming, ru*
|
||||
1.  [Revelation (https://rev.mail.ru)](https://rev.mail.ru)*: top 100M, forum, gaming, ru*
|
||||
1.  [Tanks (https://tanks.mail.ru)](https://tanks.mail.ru)*: top 100M, forum, gaming, ru*
|
||||
1.  [Warface (https://wf.mail.ru)](https://wf.mail.ru)*: top 100M, forum, ru*
|
||||
1.  [Revelation (https://rev.mail.ru)](https://rev.mail.ru)*: top 100M, forum, gaming, ru*, search is disabled
|
||||
1.  [Tanks (https://tanks.mail.ru)](https://tanks.mail.ru)*: top 100M, forum, gaming, ru*, search is disabled
|
||||
1.  [Warface (https://wf.mail.ru)](https://wf.mail.ru)*: top 100M, forum, ru*, search is disabled
|
||||
1.  [PeopleIgn (https://people.ign.com/)](https://people.ign.com/)*: top 100M, gaming, us*
|
||||
1.  [Envato (https://forums.envato.com)](https://forums.envato.com)*: top 100M, au, forum, in*
|
||||
1.  [2berega.spb.ru (https://2berega.spb.ru)](https://2berega.spb.ru)*: top 100M, ru*
|
||||
@@ -1272,7 +1272,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [Freelance.habr (https://freelance.habr.com/)](https://freelance.habr.com/)*: top 100M, freelance, ru*
|
||||
1.  [Toster (https://qna.habr.com/)](https://qna.habr.com/)*: top 100M, coding, ru*
|
||||
1.  [bbs.boingboing.net (https://bbs.boingboing.net)](https://bbs.boingboing.net)*: top 100M, forum, us*
|
||||
1.  [Gorod.dp.ua (https://forum.gorod.dp.ua/)](https://forum.gorod.dp.ua/)*: top 100M, de, forum, ua*
|
||||
1.  [Gorod.dp.ua (https://forum.gorod.dp.ua/)](https://forum.gorod.dp.ua/)*: top 100M, de, forum, ua*, search is disabled
|
||||
1.  [promalp.dp.ua (http://promalp.dp.ua)](http://promalp.dp.ua)*: top 100M*, search is disabled
|
||||
1.  [community.brave.com (https://community.brave.com)](https://community.brave.com)*: top 100M, forum, us*
|
||||
1.  [XDA (https://forum.xda-developers.com)](https://forum.xda-developers.com)*: top 100M, apps, forum*, search is disabled
|
||||
@@ -1863,7 +1863,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [forums.visual-paradigm.com (https://forums.visual-paradigm.com)](https://forums.visual-paradigm.com)*: top 100M, forum, in*
|
||||
1.  [forums.battlefield.com (https://forums.battlefield.com)](https://forums.battlefield.com)*: top 100M, forum, gaming, gb, us*, search is disabled
|
||||
1.  [social.tchncs.de (https://social.tchncs.de/)](https://social.tchncs.de/)*: top 100M, de, in*
|
||||
1.  [alliedmods (https://forums.alliedmods.net/)](https://forums.alliedmods.net/)*: top 100M, forum, gb, in, jp, tr, us, uz*
|
||||
1.  [alliedmods (https://forums.alliedmods.net/)](https://forums.alliedmods.net/)*: top 100M, forum, gb, in, jp, tr, us, uz*, search is disabled
|
||||
1.  [GameRevolution (https://forums.gamerevolution.com)](https://forums.gamerevolution.com)*: top 100M, forum, gaming, us*
|
||||
1.  [Pathofexile (https://ru.pathofexile.com)](https://ru.pathofexile.com)*: top 100M, ru, us*
|
||||
1.  [boards.theforce.net (https://boards.theforce.net)](https://boards.theforce.net)*: top 100M*
|
||||
@@ -1903,7 +1903,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [7ya (https://blog.7ya.ru)](https://blog.7ya.ru)*: top 100M, ru*
|
||||
1.  [club.7ya.ru (https://club.7ya.ru)](https://club.7ya.ru)*: top 100M, ru*
|
||||
1.  [TalkDrugabuse (https://talk.drugabuse.com)](https://talk.drugabuse.com)*: top 100M, us*
|
||||
1.  [goha (https://forums.goha.ru)](https://forums.goha.ru)*: top 100M, forum, ru*
|
||||
1.  [goha (https://forums.goha.ru)](https://forums.goha.ru)*: top 100M, forum, ru*, search is disabled
|
||||
1.  [Insanejournal (insanejournal.com)](insanejournal.com)*: top 100M, us*
|
||||
1.  [Bookmate (https://ru.bookmate.com)](https://ru.bookmate.com)*: top 100M, ru*, search is disabled
|
||||
1.  [community.startupnation.com (https://community.startupnation.com)](https://community.startupnation.com)*: top 100M, business*
|
||||
@@ -2032,7 +2032,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [QuestionableQuesting (https://forum.questionablequesting.com)](https://forum.questionablequesting.com)*: top 100M, forum, gb, jp, us*, search is disabled
|
||||
1.  [forums.sailboatowners.com (http://forums.sailboatowners.com)](http://forums.sailboatowners.com)*: top 100M, forum, us*, search is disabled
|
||||
1.  [fanat1k (https://forum.fanat1k.ru)](https://forum.fanat1k.ru)*: top 100M, forum, ru*, search is disabled
|
||||
1.  [forum.exkavator.ru (https://forum.exkavator.ru)](https://forum.exkavator.ru)*: top 100M, forum, ru*
|
||||
1.  [forum.exkavator.ru (https://forum.exkavator.ru)](https://forum.exkavator.ru)*: top 100M, forum, ru*, search is disabled
|
||||
1.  [m.smutty.com (https://m.smutty.com)](https://m.smutty.com)*: top 100M, erotic, us*, search is disabled
|
||||
1.  [community.endlessos.com (https://community.endlessos.com)](https://community.endlessos.com)*: top 100M, forum, us*
|
||||
1.  [Golangbridge (https://forum.golangbridge.org/)](https://forum.golangbridge.org/)*: top 100M, forum, in, sa, ua, us, vn*
|
||||
@@ -2194,7 +2194,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [d3.ru ()]()*: top 100M*
|
||||
1.  [Dangerousthings.com ()]()*: top 100M*
|
||||
1.  [Demotywatory ()]()*: top 100M*
|
||||
1.  [Dojoverse ()]()*: top 100M*
|
||||
1.  [Dojoverse (https://dojoverse.com)](https://dojoverse.com)*: top 100M*
|
||||
1.  [Dalnoboi (https://www.dalnoboi.ru)](https://www.dalnoboi.ru)*: top 100M, ru*
|
||||
1.  [Darkside (https://darkside.black)](https://darkside.black)*: top 100M, forum, ru*, search is disabled
|
||||
1.  [Dcpg (https://dcpg.ru/)](https://dcpg.ru/)*: top 100M, ru, ua*
|
||||
@@ -2249,7 +2249,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [Finanzfrage ()]()*: top 100M*
|
||||
1.  [Finforum (https://finforum.net)](https://finforum.net)*: top 100M, forum, ru, us, vn*
|
||||
1.  [Fireworktv (https://fireworktv.com)](https://fireworktv.com)*: top 100M, in, jp*, search is disabled
|
||||
1.  [Flbord (https://flbord.com)](https://flbord.com)*: top 100M, ru, ua*
|
||||
1.  [Flbord (https://flbord.com)](https://flbord.com)*: top 100M, ru, ua*, search is disabled
|
||||
1.  [Fm-forum (https://fm-forum.ru)](https://fm-forum.ru)*: top 100M, forum, ru*, search is disabled
|
||||
1.  [Forum.glow-dm.ru (http://forum.glow-dm.ru)](http://forum.glow-dm.ru)*: top 100M, forum, ru*
|
||||
1.  [Forum.jambox.ru (https://forum.jambox.ru)](https://forum.jambox.ru)*: top 100M, forum, ru*
|
||||
@@ -2269,7 +2269,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [FurryFandom (https://furry-fandom.ru/)](https://furry-fandom.ru/)*: top 100M, ru*, search is disabled
|
||||
1.  [G2g.com ()]()*: top 100M*
|
||||
1.  [G-news (https://g-news.com.ua)](https://g-news.com.ua)*: top 100M, in, ua*, search is disabled
|
||||
1.  [GDProfiles (https://gdprofiles.com/)](https://gdprofiles.com/)*: top 100M*
|
||||
1.  [GDProfiles (https://gdprofiles.com/)](https://gdprofiles.com/)*: top 100M*, search is disabled
|
||||
1.  [GGIZI (https://gg-izi.ru/)](https://gg-izi.ru/)*: top 100M, ru*
|
||||
1.  [GPS-Forum (http://www.gps-forum.ru)](http://www.gps-forum.ru)*: top 100M, forum, ru*
|
||||
1.  [Twitter Shadowban (https://shadowban.eu)](https://shadowban.eu)*: top 100M, jp, sa*, search is disabled
|
||||
@@ -2331,7 +2331,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [Kotburger ()]()*: top 100M*
|
||||
1.  [Kriptom (https://www.kriptom.com)](https://www.kriptom.com)*: top 100M, tr*
|
||||
1.  [KristallovNet (https://forum.kristallov.net)](https://forum.kristallov.net)*: top 100M, forum, ru*
|
||||
1.  [Lesswrong ()]()*: top 100M*
|
||||
1.  [Lesswrong (https://www.lesswrong.com)](https://www.lesswrong.com)*: top 100M*
|
||||
1.  [Letsbeef (https://www.letsbeef.com)](https://www.letsbeef.com)*: top 100M, us, vi*
|
||||
1.  [Letschatlove (https://letschatlove.com)](https://letschatlove.com)*: top 100M, in*, search is disabled
|
||||
1.  [LibReviews (https://lib.reviews)](https://lib.reviews)*: top 100M*
|
||||
@@ -2369,12 +2369,12 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [Megamodels.pl ()]()*: top 100M*
|
||||
1.  [Megane2 (http://megane2.ru/)](http://megane2.ru/)*: top 100M, ru*
|
||||
1.  [ProtonMail ()]()*: top 100M, us*
|
||||
1.  [Microchip (http://www.microchip.su)](http://www.microchip.su)*: top 100M, forum, ru*
|
||||
1.  [Microchip (http://www.microchip.su)](http://www.microchip.su)*: top 100M, forum, ru*, search is disabled
|
||||
1.  [Minecraftlist ()]()*: top 100M*
|
||||
1.  [Miped (https://miped.ru)](https://miped.ru)*: top 100M, ru*
|
||||
1.  [Mistrzowie ()]()*: top 100M*
|
||||
1.  [Mixlr (http:/mixlr.com/)](http:/mixlr.com/)*: top 100M, gb*
|
||||
1.  [Mobile-files (https://www.mobile-files.com/)](https://www.mobile-files.com/)*: top 100M, forum, ru, us*
|
||||
1.  [Mobile-files (https://www.mobile-files.com/)](https://www.mobile-files.com/)*: top 100M, forum, ru, us*, search is disabled
|
||||
1.  [Mobrep (https://www.mobrep.ru)](https://www.mobrep.ru)*: top 100M, ru*
|
||||
1.  [Money-talk (http://www.money-talk.org)](http://www.money-talk.org)*: top 100M, in, ua*, search is disabled
|
||||
1.  [Monkingme (https://www.monkingme.com/)](https://www.monkingme.com/)*: top 100M, es, in, ir*, search is disabled
|
||||
@@ -2455,7 +2455,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [Rumbleuser ()]()*: top 100M*
|
||||
1.  [Runescape ()]()*: top 100M*
|
||||
1.  [SakhalinName (https://sakhalin.name/)](https://sakhalin.name/)*: top 100M, ru*
|
||||
1.  [Salon24.pl ()]()*: top 100M*
|
||||
1.  [Salon24.pl (https://www.salon24.pl)](https://www.salon24.pl)*: top 100M*
|
||||
1.  [Saracartershow ()]()*: top 100M*
|
||||
1.  [SatsisInfo (https://satsis.info/)](https://satsis.info/)*: top 100M, ru, ua*
|
||||
1.  [Schlock (https://schlock.ru/)](https://schlock.ru/)*: top 100M, ru*
|
||||
@@ -2498,7 +2498,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [Suzuri.jp ()]()*: top 100M*, search is disabled
|
||||
1.  [Swapd ()]()*: top 100M*
|
||||
1.  [SwimmingForum (http://forumswimming.ru)](http://forumswimming.ru)*: top 100M, forum, ru*
|
||||
1.  [Syktforum (http://syktforum.ru)](http://syktforum.ru)*: top 100M, forum, ru*
|
||||
1.  [Syktforum (http://syktforum.ru)](http://syktforum.ru)*: top 100M, forum, ru*, search is disabled
|
||||
1.  [SyktyvkarOnline (http://syktyvkar-online.ru)](http://syktyvkar-online.ru)*: top 100M, ru*, search is disabled
|
||||
1.  [Szerokikadr.pl ()]()*: top 100M*
|
||||
1.  [Szmer.info ()]()*: top 100M*
|
||||
@@ -2516,7 +2516,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [Talks.by (https://talks.by)](https://talks.by)*: top 100M, by, forum*, search is disabled
|
||||
1.  [Terminator (http://terminator-scc.net.ru)](http://terminator-scc.net.ru)*: top 100M, ru*
|
||||
1.  [Thedaftclub (https://www.thedaftclub.com)](https://www.thedaftclub.com)*: top 100M, us*, search is disabled
|
||||
1.  [Thephysicsforum (https://www.thephysicsforum.com)](https://www.thephysicsforum.com)*: top 100M, forum*
|
||||
1.  [Thephysicsforum (https://www.thephysicsforum.com)](https://www.thephysicsforum.com)*: top 100M, forum*, search is disabled
|
||||
1.  [TikTok Online Viewer (https://ttonlineviewer.com)](https://ttonlineviewer.com)*: top 100M, us*, search is disabled
|
||||
1.  [Tkgr (http://tkgr.ru/)](http://tkgr.ru/)*: top 100M, ru*
|
||||
1.  [Torrent-soft (https://torrent-soft.net)](https://torrent-soft.net)*: top 100M, ru*
|
||||
@@ -2922,13 +2922,13 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [lubuntu.ru (https://lubuntu.ru)](https://lubuntu.ru)*: top 100M, forum*
|
||||
1.  [forum.dusterclub.ru (http://forum.dusterclub.ru)](http://forum.dusterclub.ru)*: top 100M, forum*, search is disabled
|
||||
1.  [openssource.info (https://openssource.info)](https://openssource.info)*: top 100M, forum, ru*
|
||||
1.  [rub.altai.su (http://rub.altai.su)](http://rub.altai.su)*: top 100M, forum, ru*
|
||||
1.  [rub.altai.su (http://rub.altai.su)](http://rub.altai.su)*: top 100M, forum, ru*, search is disabled
|
||||
1.  [mailpass.site (https://mailpass.site)](https://mailpass.site)*: top 100M, forum, ru*
|
||||
1.  [forum.rastrnet.ru (http://forum.rastrnet.ru)](http://forum.rastrnet.ru)*: top 100M, forum, ru*
|
||||
1.  [forumbusiness.net (http://forumbusiness.net)](http://forumbusiness.net)*: top 100M, forum*, search is disabled
|
||||
1.  [se.guru (https://se.guru)](https://se.guru)*: top 100M, forum, ru*
|
||||
1.  [ovnl.in (https://ovnl.in)](https://ovnl.in)*: top 100M, forum*, search is disabled
|
||||
1.  [porschec.ru (http://porschec.ru)](http://porschec.ru)*: top 100M, forum*
|
||||
1.  [porschec.ru (http://porschec.ru)](http://porschec.ru)*: top 100M, forum*, search is disabled
|
||||
1.  [mindmachine.ru (https://mindmachine.ru/)](https://mindmachine.ru/)*: top 100M, forum*
|
||||
1.  [nelubit.ru (https://nelubit.ru)](https://nelubit.ru)*: top 100M, forum*
|
||||
1.  [actual-porn.org (http://actual-porn.org)](http://actual-porn.org)*: top 100M, forum*, search is disabled
|
||||
@@ -2955,7 +2955,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [getmakerlog.com (https://getmakerlog.com)](https://getmakerlog.com)*: top 100M, business*
|
||||
1.  [aussiehomebrewer.com (https://aussiehomebrewer.com)](https://aussiehomebrewer.com)*: top 100M*
|
||||
1.  [forum-ukraina.net (https://forum-ukraina.net)](https://forum-ukraina.net)*: top 100M*
|
||||
1.  [forum-history.ru (http://forum-history.ru)](http://forum-history.ru)*: top 100M*
|
||||
1.  [forum-history.ru (http://forum-history.ru)](http://forum-history.ru)*: top 100M*, search is disabled
|
||||
1.  [forum.alconar.ru (https://forum.alconar.ru)](https://forum.alconar.ru)*: top 100M*
|
||||
1.  [forum.lancerx.ru (https://forum.lancerx.ru)](https://forum.lancerx.ru)*: top 100M*
|
||||
1.  [mfarmer.ru (http://www.mfarmer.ru)](http://www.mfarmer.ru)*: top 100M*, search is disabled
|
||||
@@ -3038,7 +3038,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [dnd-wiki.org ()]()*: top 100M*
|
||||
1.  [dogcraft.net ()]()*: top 100M*
|
||||
1.  [elixirforum.com ()]()*: top 100M*
|
||||
1.  [en.brickimedia.org ()]()*: top 100M*
|
||||
1.  [en.brickimedia.org (https://en.brickimedia.org)](https://en.brickimedia.org)*: top 100M*
|
||||
1.  [en.illogicopedia.org ()]()*: top 100M*
|
||||
1.  [en.uncyclopedia.co ()]()*: top 100M*
|
||||
1.  [en.wikifur.com ()]()*: top 100M*
|
||||
@@ -3240,7 +3240,7 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [www.gta-multiplayer.cz (https://www.gta-multiplayer.cz)](https://www.gta-multiplayer.cz)*: top 100M*
|
||||
1.  [Blitz Tactics (https://blitztactics.com/)](https://blitztactics.com/)*: top 100M*
|
||||
1.  [Breach Sta.rs Forum (https://breachsta.rs/)](https://breachsta.rs/)*: top 100M, forum*
|
||||
1.  [Codolio (https://codolio.com/)](https://codolio.com/)*: top 100M*
|
||||
1.  [Codolio (https://codolio.com/)](https://codolio.com/)*: top 100M*, search is disabled
|
||||
1.  [Coinvote (https://coinvote.cc/)](https://coinvote.cc/)*: top 100M*
|
||||
1.  [Cracked Forum (https://cracked.sh/)](https://cracked.sh/)*: top 100M, forum*
|
||||
1.  [Discord.bio (https://discord.bio/)](https://discord.bio/)*: top 100M*
|
||||
@@ -3258,18 +3258,18 @@ Rank data fetched from Majestic Million by domains.
|
||||
1.  [PentesterLab (https://pentesterlab.com/)](https://pentesterlab.com/)*: top 100M, hacking, us*
|
||||
1.  [CTFtime (https://ctftime.org/)](https://ctftime.org/)*: top 100M, hacking, us*
|
||||
|
||||
The list was updated at (2026-03-26)
|
||||
The list was updated at (2026-03-27)
|
||||
## Statistics
|
||||
|
||||
Enabled/total sites: 2666/3254 = 81.93%
|
||||
Enabled/total sites: 2632/3254 = 80.89%
|
||||
|
||||
Incomplete message checks: 374/2666 = 14.03% (false positive risks)
|
||||
Incomplete message checks: 344/2632 = 13.07% (false positive risks)
|
||||
|
||||
Status code checks: 639/2666 = 23.97% (false positive risks)
|
||||
Status code checks: 636/2632 = 24.16% (false positive risks)
|
||||
|
||||
False positive risk (total): 38.00%
|
||||
False positive risk (total): 37.23%
|
||||
|
||||
Sites with probing: 500px, Aparat (disabled), Armchairgm, BinarySearch (disabled), BleachFandom, BongaCams, BuyMeACoffee, Cent, Chess, Code Sandbox, Discord, Diskusjon.no, Disqus, Docker Hub, Duolingo, FandomCommunityCentral, Gab, GitHub, GitLab, Google Plus (archived), Gravatar, Holopin, Imgur, Issuu, Keybase, Kick, Kvinneguiden, Livejasmin, LocalCryptos (disabled), MicrosoftLearn, MixCloud, Monkeytype, Niftygateway, Omg.lol, Picsart, Rarible, Reddit, Reddit Search (Pushshift) (disabled), RoyalCams, SOOP, SportsTracker, Spotify (disabled), Substack, TAP'D, Topcoder, Trello, Twitch, Twitter, Twitter Shadowban (disabled), UnstoppableDomains, Vimeo, Warframe Market, Weibo, Yapisal (disabled), YouNow, hashnode, nightbot, notabug.org, polarsteps, qiwi.me (disabled)
|
||||
Sites with probing: 500px, Aparat (disabled), Armchairgm, BinarySearch (disabled), BleachFandom, BongaCams, BuyMeACoffee, Cent, Chess, Code Sandbox, Discord, Diskusjon.no, Disqus, Docker Hub, Duolingo, FandomCommunityCentral, Gab, GitHub, GitLab, Google Plus (archived), Gravatar, Holopin, Imgur, Issuu, Keybase, Kick, Kvinneguiden, Lesswrong, Livejasmin, LocalCryptos (disabled), MicrosoftLearn, MixCloud, Monkeytype, Niftygateway, Omg.lol, Picsart, Rarible, Reddit, Reddit Search (Pushshift) (disabled), RoyalCams, SOOP, SportsTracker, Spotify (disabled), Substack, TAP'D, Topcoder, Trello, Twitch, Twitter, Twitter Shadowban (disabled), UnstoppableDomains, Vimeo, Warframe Market, Weibo, Yapisal (disabled), YouNow, en.brickimedia.org, hashnode, nightbot, notabug.org, polarsteps, qiwi.me (disabled)
|
||||
|
||||
Sites with activation: Spotify (disabled), Twitter, Vimeo, Weibo
|
||||
|
||||
|
||||
+67
-9
@@ -26,6 +26,7 @@ sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
try:
|
||||
import aiohttp
|
||||
from yarl import URL as YarlURL
|
||||
except ImportError:
|
||||
print("aiohttp not installed. Run: pip install aiohttp")
|
||||
sys.exit(1)
|
||||
@@ -74,8 +75,14 @@ def color(text: str, c: str) -> str:
|
||||
|
||||
|
||||
async def check_url_aiohttp(url: str, headers: dict = None, follow_redirects: bool = True,
|
||||
timeout: int = 15, ssl_verify: bool = False) -> dict:
|
||||
"""Check a URL using aiohttp and return detailed response info."""
|
||||
timeout: int = 15, ssl_verify: bool = False,
|
||||
method: str = "GET", payload: dict = None) -> dict:
|
||||
"""Check a URL using aiohttp and return detailed response info.
|
||||
|
||||
Args:
|
||||
method: HTTP method ("GET" or "POST").
|
||||
payload: JSON payload for POST requests (dict, will be serialized).
|
||||
"""
|
||||
headers = headers or DEFAULT_HEADERS.copy()
|
||||
result = {
|
||||
"method": "aiohttp",
|
||||
@@ -96,7 +103,14 @@ async def check_url_aiohttp(url: str, headers: dict = None, follow_redirects: bo
|
||||
timeout_obj = aiohttp.ClientTimeout(total=timeout)
|
||||
|
||||
async with aiohttp.ClientSession(connector=connector, timeout=timeout_obj) as session:
|
||||
async with session.get(url, headers=headers, allow_redirects=follow_redirects) as resp:
|
||||
# Use encoded=True if URL contains percent-encoded chars to prevent double-encoding
|
||||
request_url = YarlURL(url, encoded=True) if '%' in url else url
|
||||
request_kwargs = dict(headers=headers, allow_redirects=follow_redirects)
|
||||
if method.upper() == "POST" and payload is not None:
|
||||
request_kwargs["json"] = payload
|
||||
|
||||
request_fn = session.post if method.upper() == "POST" else session.get
|
||||
async with request_fn(request_url, **request_kwargs) as resp:
|
||||
result["status"] = resp.status
|
||||
result["final_url"] = str(resp.url)
|
||||
|
||||
@@ -438,21 +452,54 @@ async def diagnose_site(site_config: dict, site_name: str) -> dict:
|
||||
print(f" {color('[!]', Colors.RED)} No usernameClaimed defined")
|
||||
return diagnosis
|
||||
|
||||
# Build full URL
|
||||
# Build full URL (display URL)
|
||||
url_template = url.replace("{urlMain}", url_main).replace("{urlSubpath}", site_config.get("urlSubpath", ""))
|
||||
|
||||
# Build probe URL (what Maigret actually requests)
|
||||
url_probe = site_config.get("urlProbe", "")
|
||||
if url_probe:
|
||||
probe_template = url_probe.replace("{urlMain}", url_main).replace("{urlSubpath}", site_config.get("urlSubpath", ""))
|
||||
else:
|
||||
probe_template = url_template
|
||||
|
||||
# Detect request method and payload
|
||||
request_method = site_config.get("requestMethod", "GET").upper()
|
||||
request_payload_template = site_config.get("requestPayload")
|
||||
|
||||
headers = DEFAULT_HEADERS.copy()
|
||||
# For API probes (urlProbe, POST), use neutral Accept header instead of text/html
|
||||
# which can cause servers to return HTML instead of JSON
|
||||
if url_probe or request_method == "POST":
|
||||
headers["Accept"] = "*/*"
|
||||
if site_config.get("headers"):
|
||||
headers.update(site_config["headers"])
|
||||
|
||||
if url_probe:
|
||||
print(f" urlProbe: {url_probe}")
|
||||
if request_method != "GET":
|
||||
print(f" requestMethod: {request_method}")
|
||||
if request_payload_template:
|
||||
print(f" requestPayload: {request_payload_template}")
|
||||
|
||||
# 2. Connectivity test
|
||||
print(f"\n--- {color('2. CONNECTIVITY TEST', Colors.BOLD)} ---")
|
||||
url_claimed = url_template.replace("{username}", claimed)
|
||||
url_unclaimed = url_template.replace("{username}", unclaimed)
|
||||
probe_claimed = probe_template.replace("{username}", claimed)
|
||||
probe_unclaimed = probe_template.replace("{username}", unclaimed)
|
||||
|
||||
# Build payloads with username substituted
|
||||
payload_claimed = None
|
||||
payload_unclaimed = None
|
||||
if request_payload_template and request_method == "POST":
|
||||
payload_claimed = json.loads(
|
||||
json.dumps(request_payload_template).replace("{username}", claimed)
|
||||
)
|
||||
payload_unclaimed = json.loads(
|
||||
json.dumps(request_payload_template).replace("{username}", unclaimed)
|
||||
)
|
||||
|
||||
result_claimed, result_unclaimed = await asyncio.gather(
|
||||
check_url_aiohttp(url_claimed, headers),
|
||||
check_url_aiohttp(url_unclaimed, headers)
|
||||
check_url_aiohttp(probe_claimed, headers, method=request_method, payload=payload_claimed),
|
||||
check_url_aiohttp(probe_unclaimed, headers, method=request_method, payload=payload_unclaimed)
|
||||
)
|
||||
|
||||
print(f" Claimed ({claimed}): status={result_claimed['status']}, error={result_claimed['error']}")
|
||||
@@ -523,7 +570,18 @@ async def diagnose_site(site_config: dict, site_name: str) -> dict:
|
||||
diagnosis["warnings"].append(f"absenceStrs not found in unclaimed page")
|
||||
print(f" {color('[WARN]', Colors.YELLOW)} absenceStrs not found in unclaimed page")
|
||||
|
||||
if presense_found_claimed and not absence_found_claimed and absence_found_unclaimed:
|
||||
# Check works if: claimed is detected as present AND unclaimed is detected as absent.
|
||||
# Presence detection: presenseStrs found (or empty = always true).
|
||||
# Absence detection: absenceStrs found in unclaimed (or empty = never, rely on presenseStrs only).
|
||||
# With only presenseStrs: works if found in claimed but NOT in unclaimed.
|
||||
# With only absenceStrs: works if found in unclaimed but NOT in claimed.
|
||||
# With both: standard combination.
|
||||
claimed_is_present = presense_found_claimed and not absence_found_claimed
|
||||
unclaimed_is_absent = (
|
||||
(absence_strs and absence_found_unclaimed) or
|
||||
(presense_strs and not presense_found_unclaimed)
|
||||
)
|
||||
if claimed_is_present and unclaimed_is_absent:
|
||||
print(f" {color('[OK]', Colors.GREEN)} Message check should work correctly")
|
||||
diagnosis["working"] = True
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ This module generates the listing of supported sites in file `SITES.md`
|
||||
and pretty prints file with sites data.
|
||||
"""
|
||||
import sys
|
||||
import socket
|
||||
import requests
|
||||
import logging
|
||||
import threading
|
||||
@@ -64,6 +65,49 @@ def get_base_domain(url):
|
||||
return ""
|
||||
|
||||
|
||||
def check_dns(domain, timeout=5):
|
||||
"""Check if a domain resolves via DNS. Returns True if it resolves."""
|
||||
try:
|
||||
socket.setdefaulttimeout(timeout)
|
||||
socket.getaddrinfo(domain, None)
|
||||
return True
|
||||
except (socket.gaierror, socket.timeout, OSError):
|
||||
return False
|
||||
|
||||
|
||||
def check_sites_dns(sites):
|
||||
"""Check DNS resolution for all sites. Returns a set of site names that failed."""
|
||||
SKIP_TLDS = ('.onion', '.i2p')
|
||||
domains = {}
|
||||
for site in sites:
|
||||
domain = get_base_domain(site.url_main)
|
||||
if domain and not any(domain.endswith(tld) for tld in SKIP_TLDS):
|
||||
domains.setdefault(domain, []).append(site)
|
||||
|
||||
failed_sites = set()
|
||||
results = {}
|
||||
|
||||
def resolve(domain):
|
||||
results[domain] = check_dns(domain)
|
||||
|
||||
threads = []
|
||||
for domain in domains:
|
||||
t = threading.Thread(target=resolve, args=(domain,))
|
||||
threads.append(t)
|
||||
t.start()
|
||||
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
for domain, resolved in results.items():
|
||||
if not resolved:
|
||||
for site in domains[domain]:
|
||||
failed_sites.add(site.name)
|
||||
logging.warning(f"DNS resolution failed for {domain}")
|
||||
|
||||
return failed_sites
|
||||
|
||||
|
||||
def get_step_rank(rank):
|
||||
def get_readable_rank(r):
|
||||
return RANKS[str(r)]
|
||||
@@ -86,6 +130,8 @@ def main():
|
||||
parser.add_argument('--empty-only', help='update only sites without rating', action='store_true')
|
||||
parser.add_argument('--exclude-engine', help='do not update score with certain engine',
|
||||
action="append", dest="exclude_engine_list", default=[])
|
||||
parser.add_argument('--dns-check', help='disable sites whose domains do not resolve via DNS',
|
||||
action='store_true')
|
||||
|
||||
pool = list()
|
||||
|
||||
@@ -103,6 +149,24 @@ Rank data fetched from Majestic Million by domains.
|
||||
|
||||
""")
|
||||
|
||||
if args.dns_check:
|
||||
print("Checking DNS resolution for all site domains...")
|
||||
failed = check_sites_dns(sites_subset)
|
||||
disabled_count = 0
|
||||
re_enabled_count = 0
|
||||
for site in sites_subset:
|
||||
if site.name in failed:
|
||||
if not site.disabled:
|
||||
site.disabled = True
|
||||
disabled_count += 1
|
||||
print(f" Disabled {site.name}: DNS does not resolve ({get_base_domain(site.url_main)})")
|
||||
else:
|
||||
if site.disabled:
|
||||
# Re-enable previously disabled site if DNS now resolves
|
||||
# (only if it was likely disabled due to DNS failure)
|
||||
pass
|
||||
print(f"DNS check complete: {disabled_count} site(s) disabled, {len(failed)} domain(s) unresolvable.")
|
||||
|
||||
majestic_ranks = {}
|
||||
if args.with_rank:
|
||||
majestic_ranks = fetch_majestic_million()
|
||||
|
||||
Reference in New Issue
Block a user