[plug] A plan for spam spiders.

[Shayne O'Neill]

One of the things that struck me while contemplating spam and the somewhat
clever teergrubing technique is that these ideas are never applied at the
coalfront of spam crime. The collection of email adresses. Web spiders
seem to generate a fantastic amount of traffic, and when combined with
dynamic pages on a web server, quite alot of webserver load.

I intend to fight back.

There are 2 types of spider. Good ones and bad ones. Good spiders , like
google will read robots.txt , steer clear of directories the webmaster
doesnt want spidered and importantly respect crawl delay settings asking
it to mellow out.

SPam spiders however , I have observed, do not, they tend to bombard a
webserver, sometimes to its knees, and completely ignore crawl-delay.

This is the signature of a spam spider.

I also imagine these spiders to be exceedingly poorly written.

What would happen if we configured our server, we set a trap that would
lead a spam spider to a 'special' page, while warning off legit spiders
via robots.txt.

My plan is that said page would then deliver a gzipped file consisting of
one gigabyte of zero's to the spider gzipped down to a couple of hundred
kilobytes. and sent identified as a gzipped html page.

So when the spider finds a link to "DO NOT CLICK ME AS THIS PAGE WILL
CRASH YOUR COMPUTER" which is also enticingly placed in robots.txt as
forbidden fruit, it excitedly clicks through, recieves a gzipped html
file, which it unpacks to view the hidden goodies, and BLAM! 1 gigabit of
crud explodes in its head, depleting the spam servers memory, and vmem and
causing the smoke to leak out of its vile little brain.

The question is;- WOULD IT WORK!

Shayne.

--
 I wish a robot would get elected president. That way, when he came to
town, we could all take a shot at him and not feel too bad.
- Jack Handey (And now, Deep thoughts)