[plug] logging incomplete downloads in Apache

[Jay Turner]

> On Mon, Apr 19, 2004 at 12:01:16AM +0800, Craig Ringer wrote:
> > I don't know how many of those are prematurely terminated.
>
> In addition to the other comments already made in this thread, I'm just
> going to add that proxies make this a lot harder to work out.  A proxy
> will often grab the entire contents of a file, but the user may abort
> the download before *they* get it.
>
> This is, unfortunately, rather impossible to detect.

The other thing that makes this process even harder I have just discovered
is download managers/accelerators.

They cause all sorts of havoc in Apaches logs as it appears that the manager
first gets the entire file size (which is recorded as a hit in the log) and
then launches several connections to the file each downloading a portion of
the file to produce a full download. The problem is that when you run
something like webalizer over the log, it records all the information
(including the initial request for the file size) making it appear that
twice as much data had been downloaded than actually had.

Due to this, so far this month it looks like over 200Gb has been downloaded
from our webserver.
This obviously is not the case, but it makes our Webalizer kb stats useless.

Jay