>From: Innfinit@ix.netcom.com (Captain Infinity)
>
>> Beware of using Deja News for any serious statistical data collection.
>> They are notoriously inaccurate.
>
>in what way are 'they' inaccurate?
Just one quick and dirty example:
I recently did a data collection for articles posted to alt.fan.tom-servo.
This was problematical for several reasons, the most significant of which was
the fact that Deja News did not begin archiving AFT-S until 11-24-97.
(I first noticed the lack of AFT-S articles in the Deja News database in
mid-October. My request for info was answered by User Liaison 6
<help6@dejanews.com>. He told me this:
>Deja News only archives the newsgroups that are carried by our newsfeed.
>Unfortunately, what newsgroups they carry is beyond our control. The
>newsgroup you mention may not be carried by our newsfeed because it is a
>new newsgroup or recently released to the public. In this case, it may
>take a while for our newsfeed to pick it up. Rest assured, when it does
>pick it up, we'll archive it!
When I informed him that AFT-S had been newgrouped on 7-15-94, three and
one-half years ago, there was no answer...but a few weeks later they began
archiving the group.)
BTW, let me clarify what I mean by "archiving". If you do a filtered search
on AFT-S you *will* find posts on Deja News prior to 11-24-97. However, they
are all posts that were crossposted to at least one other group which Deja
News was archiving. No posts sent *only* to AFT-S prior to 11-24-97 can be
found on Deja News.
So in this single case, statistical data collection on a group is impossible
for any data prior to 11-24-97, about 3.5 years after the group was created.
As I said, I read and downloaded (and saved) everything on that group since
10-5-97 (including spam). My archive shows the following:
10-5 to 11-23 727 posts
11-24 to 12-31 922 posts
total, 10-5 to 12-31 1649 posts
Deja News' numbers, using a filter on group: alt.fan.tom-servo and setting the
date ranges as required, show the following:
10-5-97 to 11-23 575 posts
11-24 to 12-31 967 posts
total, 10-5 to 12-31 1542 posts
The wide variances in numbers during the different date ranges, plus the final
difference of 107 posts (in just a three month period for a newsgroup that has
existed since 7-15-94) makes me sceptical as to how accurate Deja News can
possibly be, in general. Also, take into consideration the fact that Deja
News breaks large articles into segments, artificially pumping up the message
count. (Cancel messages also increase the numbers.) I know for a fact that
at least 18 of the messages in Deja News' count for AFT-S from 11-24 to 12-31
are spurious; I posted the AFT-S FQA to that group three times in that period,
and Deja News broke it into seven separate articles each time.
On the whole, Deja News is a fantastic resource. I use it regularly and am
damn glad it exists. But I wouldn't settle any bets with it.
Hmmm. Okay, so it wasn't "quick and dirty" like I promised. Oh well. I get
long winded, sometimes. :-)
HTH.
Best,
Captain Infinity
...maintainer of the Frequently Questioned Answers file for alt.fan.tom-servo