by Mike Shea on 2 August 2005
Yesterday I received a bill from pair.com, the host that hosts all five of my websites. The $100 bill was for bandwidth overage charges. On a typical month I send out a total of about 12 GB of data across all of my sites. Apparently last month I sent out over 94, far more than I am allowed. The overage charge ran $100. Before I continue I should mention that since the discovery of this great evil I am about to describe, Pair.com refunded me the $100 and I ended up paying nothing extra. I love Pair.com and they are worth every penny I pay to run these sites.
Spammers are paid to get higher search engine rankings for websites. One big way is to get links on thousands of other websites to improve their page rank. The more websites that link to a site, the higher it ranks in search engines.
This caused a surge of comment spamming for websites running Movable Type. It got so bad on MikeShea.net that I finally removed comments completely. It's not like I care what you people think anyway. On Mobhunter, however, comments are a critical element of that community. The spam got bad over there too, about 30 spammed comments a week and they were tricky to find. I added a tool called MT Close that let me shut off commenting on old articles all at once. Since spammers can't find the pages fast enough to add old comments, they don't spam nearly as often. I still get one or two a month but it's rare.
Referrer spamming is even worse than comment spamming. I used to post my website statistics including the top pages linking to me. Evil spammers found these pages and began hitting them at a rate of about 100,000 to 300,000 hits in one day. On 26 July, I received 300,000 hits for a total of 35 GB of data transferred from a single spamming machine. That's nearly three times the bandwidth I use in a month spent on a single day.
Why? So spammers could get their fake referrer URLs added to my stupid web statistics page as a top page. When I looked at my referrer log, it showed thirty porn sites directing about 20,000 hits each to my site. Of course, the referrals were fake, they never linked to me, their bot simply lied and said they had just so their links would show up on a web page. According to some articles I read, the spammers do this to about 10,000 to 50,000 websites at a time.
So think about the math for a moment. In order to add a single set of URLs, data less than 1k in size, spammers created about 80 GB worth of web traffic. That's traffic paid for by you and me both. 80 GB of traffic to add perhaps 15 URLs to a single web page on my little website. Now that is true waste.
To stop them I had to go through a variety of methods. It is easy to block IP addresses using a .htaccess file but spammers are very good at changing IPs, domains, and even the referring URLs faster than a guy like me can keep up. I took down the statistics page so they won't find anything to hit anymore to get a URL added. Unfortunately I have not yet found a way to limit the number of hits a single IP can give me in a single day. Right now I simply have to watch my logs and see. I can't even find a good way to throttle the amount of bandwidth my sites can send in a day.
Spam is a far bigger problem than we realize. I get over 12,000 spammed emails a month. Now my website had to serve 80 GB worth of data so spammers could get a better google rank. I've had to cut off communications with other people because spammers filled it with crap.
Spam has to be stopped or it will fill up the net and suffocate anything real we ever read or write.