[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

How come search engines don't crawl the list and I can't find 'robots.txt'?

To: coderman <[email protected]>
Subject: How come search engines don't crawl the list and I can't find 'robots.txt'?
From: [email protected] (Georgi Guninski)
Date: Sun, 13 Apr 2014 17:38:19 +0300
Cc: cpunks <[email protected]>
In-reply-to: <CAJVRA1Qg_PLXCStjPcWkHU4TcGNvG6m8_O79M0d9AH_nXxM5VA@mail.gmail.com>
References: <[email protected]$> <CAJVRA1Qg_PLXCStjPcWkHU4TcGNvG6m8_O79M0d9AH_nXxM5VA@mail.gmail.com>

On Sun, Apr 13, 2014 at 07:08:27AM -0700, coderman wrote:
> On Sun, Apr 13, 2014 at 6:53 AM, Georgi Guninski <[email protected]> wrote:
> > AFAICT the list archives don't show in search engines
> > and in addition don't see /robots.txt.
> >
> > How so?
> 
> back when traffic was handled on the al-qaeda.net domain, indexing by
> western search engines was considered to be "material support to
> Al-qaeda"...
> 
> now they've just forgotten to undo their filters.
> 
> ;)

lol...

the russian and chinese comrades don't show the
list too, might be for technical reasons.

does a single search engine crawls
the archives and displays the results?

Follow-Ups:
- How come search engines don't crawl the list and I can't find 'robots.txt'?
  - From: [email protected] ([email protected])

References:
- How come search engines don't crawl the list and I can't find 'robots.txt'?
  - From: [email protected] (Georgi Guninski)
- How come search engines don't crawl the list and I can't find 'robots.txt'?
  - From: [email protected] (coderman)

Prev by Date: "How I obtained the private key for www.cloudflarechallenge.com"
Next by Date: [cryptome] Re: snowman news
Previous by thread: How come search engines don't crawl the list and I can't find 'robots.txt'?
Next by thread: How come search engines don't crawl the list and I can't find 'robots.txt'?
Index(es):
- Date
- Thread