[plug] A plan for spam spiders.

[Shayne O'Neill]

Hmm. So the question here, is then how do I coax apache to serve a gz file
as gz encrypted html file? I'd like to test this.


--
 I wish a robot would get elected president. That way, when he came to
town, we could all take a shot at him and not feel too bad.
- Jack Handey (And now, Deep thoughts)

On Sat, 7 May 2005, Craig Ringer wrote:

> On Sat, 2005-05-07 at 18:13 +0800, Daniel J. Axtens wrote:
> > > So when the spider finds a link to "DO NOT CLICK ME AS THIS PAGE WILL
> > > CRASH YOUR COMPUTER" which is also enticingly placed in robots.txt as
> > > forbidden fruit, it excitedly clicks through, recieves a gzipped html
> > > file, which it unpacks to view the hidden goodies, and BLAM! 1 gigabit of
> > > crud explodes in its head, depleting the spam servers memory, and vmem and
> > > causing the smoke to leak out of its vile little brain.
> > >
> > > The question is;- WOULD IT WORK!
> >
> > Would a spam spider ungzip a gzipped file?
>
> If it uses a quality library for HTTP, then it could be done
> transparently (HTTP supports compressed results). Of course, if the bot
> and HTTP library avoid loading the whole stream into RAM at once and
> instead process it as a stream, they wouldn't much mind (except for the
> processing overhead).
>
> --
> Craig Ringer
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://www.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.linux.org.au
>