[plug] [OT] http: different results from browser and telnet

[Carl Gherardi]

On Thu, 25 Oct 2001, Mike Holland wrote:

> > On Thu, 25 Oct 2001, Carl Gherardi wrote:
> >
> > > Ok bit more of a description is in order, i'm writing a program in c which
> > > get the results of a web search by directly querying a search engine then
> > > iteratively fetching and stripping links out of resulting pages.
>
> masochist! :) Try using perl. You will have a lot less wheels to
> re-invent. Even if you have to learn perl, it will still be quicker.
> And the program could probably fit on one page.

Yeah.. maybe a little masochistic. I chose C/C++ as I dont have the
experience in that language (and it is an experience...) and perl would
have been too easy ;)

The other reason is the need for a concurrent server(browser?) for web
pages, the program will fetch anywhere up to 3000 pages per search. I felt
(probably wrongly) at the time that concurrency was better suited to C/C++

>   Perl does HTTP already, and makes it easy to parse and modify HTML
> pages. I have done similar to what you describe, if you would like some
> example code.
>

I wouldn't mind having a look at that for interests sake, if you're
feeling generous contact me off list

Thanks

Carl