[plug] [OT] http: different results from browser and telnet

[Matt Kemner]

On Thu, 25 Oct 2001, Carl Gherardi wrote:

> Ok bit more of a description is in order, i'm writing a program in c which
> get the results of a web search by directly querying a search engine then
> iteratively fetching and stripping links out of resulting pages.
>
> How can I get my page to return the same as netscape?

By making your web browser HTTP/1.1 compliant :)
(or alternatively by using an external command-line program that is
 HTTP/1.1 complaint, such as wget or curl)

At the very least you must support the "Host:" header, which is what most
webservers now use to support virtual hosts.

Basically, through the use of the Host: header, it is no longer necessary
to assign an IP address for each website you host (which is good because
IP addresses are in short supply) - but, if you connect to an IP address
and request a page and don't tell the web server which page you were
expecting, the web server will return the default page.

For example, the PLUG server has two websites on it - the main PLUG
website, and golden.wattle.id.au

If you were to telnet to "golden.wattle.id.au" on port 80 and request "/"
the web server only sees an incoming connection to it's IP address, with
no indication as to what hostname you typed in, and will return the PLUG
website instead.

By specifying the "Host:" header (as Leon pointed out) - which is in the
HTTP/1.1 standard, and which HTTP/1.0 web browsers also support, you tell
the web server which web site you were really after

ie if you telnet to PLUG's IP address (202.61.150.16) and type:
GET / HTTP/1.1
Host: golden.wattle.id.au

(press enter 2x after the Host: header to signify end of headers)

you will get the g.w.id homepage

You would get the same page if you telnetted to "plug.linux.org.au" and
specified the Host: header for g.w.id - remember that your local DNS
resolves whatever hostname you type in, into an IP address, so the
webserver does not know what you typed in there.

 - Matt