The onslaught just keeps coming. It’s on pace to easily meet 20,000 spam comments in 24 hours. I had a very small handful of false negatives, but they were easily dispatched by clicking a couple of checkboxes on an admin page.

20,000 spam attempts. Peaking at multiple attempts per second, with over 500 spam bots simultaneously spidering/spamming the site. Drupal seems to be holding its own, triggering throttles to shut down higher-load functions so the site remains responsive.

And Akismet is dutifully blocking the roaches from getting through. Their success rate just keeps dropping - down to 0.1% success rate, and even that is only temporary, until I manually remove the small handful that gets by Akismet’s watchful gaze. The spammer’s effective success rate is exactly 0% - not asymptotally approaching 0, but exactly 0. Zero. Absolute Zero. Cold heat death for spam.

I’m pretty sure they’re sticking around here because the bots think they are succeeding - Drupal accepts the comment, so the bots think it worked. What they don’t realize is that the comment is immediately unpublished by the Akismet module - as soon as the Akismet retuns its spam flag.

Thankfully, Dreamhost appears to be handling it like a champ. Not even breaking a sweat. I’m pretty sure that if this blog was still back on GoDaddy’s servers, it would have curled up into the fetal position and mumbled quietly to itself…

I'd tried Akismet before, and wound up reverting to Spam Karma 2 - actually, I think SK2 was interfering with Akismet, so that likely wasn't a fair comparison, since both were running at the same time.

But, I've been running the Drupal Akismet module for 10 days now, and it's been performing absolutely perfectly. For example, this blog has been under a sustained spam attack for the last 12 hours or so - over 400 600 spam attempts just last night (200 just while writing this post) - and not a single one of the roaches got through. I just went through the Akismet comment moderation queue to look for false positives, and there wasn't a single one. So it's batting 1000 under a significant spam attack.

Mad kudos to Akismet, and to Markus for porting it to Drupal!

Actually, I'm pretty impressed at how this blog is running under this spam attack - it's still responsive, pages load quickly, and posting new content and comments is still working. Drupal's handling the extra load without breaking a sweat.

I'lll be keeping the Akismet module running on my blog, but will keep playing with the Spam.module update on our campus server due to licensing requirements for Akismet. Our use falls under the "commercial" category, and with the number of Drupal sites we're using, the cost can't be justified (yet - maybe if we get hit by spammers that don't get blocked by spam.module we'll adjust things to find the cash).

Update: Over 2000 attempted spam comments in the last 24 hours, and every single one was stopped by Akismet. Here's a screenshot from the Akismet moderation log - I didn't have to see any of these, and they were coming at me every in bursts of up to 1 spam per second for 24 hours…

Akismet Spam Attack: A screenshot of the Akismet Drupal module moderation log, during a sustained spam attack where Akismet blocked 100% of attempted spams, with no false positives.

Update 2: It's been almost exactly 24 hours since I first wrote this entry. Since then, an additional 2000 comment spam attempts have been successfullly blocked by Akismet. And a whopping 8 spam comments got through to my blog. 8. That's it. Out of over 3000 attempts in a day and a half. That's roughly a 0.267% success rate for the spammers. But, the economics of it make even THAT a worthwhile use of their time.

I debated doing something more proactive to stop the spammers altogether, but then thought that it's probably better for them to leave their bots pointed here, getting no benefit at all, than randomly spraying their spam across the 'net and maybe hitting someone's blog that isn't using an effective spam blocker. The way the Drupal Akismet module works, the spammers think they're getting every comment posted here, but the module immediately unpublishes their spam as soon as Akismet responds. That's a pretty sane way to set up the block - don't tell the spammers that they're wasting their time, just nuke their spam without a whimper…

I've been running this blog on Drupal for a while now, and am generally quite happy with it. The one thing I'd been missing from my days powered by WordPress was a transparent and effective spam blocker. I was so totally spoiled by Spam Karma 2 that everything else just seems like a kid's toy in comparison.

I'd installed the Spam module shortly after I switched to Drupal, but it never seemed to actually block spam. It is pretty handy at removing it, but the URL and keyword matches didn't seem to actually stop spam.

Then, this morning, some spamass decided it would be fun to point his (I'm assuming this jerk is a guy) spambots at my blog. The spam was consistently getting past Spam.module, but it was pretty easy to clean up after the jerk. Still, it's no fun playing mop-up after a cretinous script kiddie, so I rolled up my sleeves to see if I could duct tape a better solution together.

Thankfully, the work had already been done for me, just not updated to Drupal 4.7.  The Spam.module for 4.6 ships with a Spam URI Realtime Blocklist module, which will check incoming comments with 6 different realtime-updated shared lists of known spammers.

So, I fired up the Form Updater module, converted the spam_surbl.module code to 4.7, and deployed it. It seems to work so far - of course I'm jynxing it now… I've attached my hack update of spam_surbl.module, which I'm using on Drupal 4.7 here. I'll send a copy to the developer of Spam.module in case he wants to include an updated blocklist module (I didn't convert the .mysql file to an .install autoinstaller, so you may need to run that manually to get the module ready). 

Our little experiment at wiki.ucalgary.ca has been having some rough times. It’s gotten so frustrating that I’d had to temporarily disable new account creation in a desparate attempt to thwart the automated spambots (which automatically create a new account for each edit so it’s harder to roll them back).

I’ve just updated the wiki to the latest and greatest Mediawiki 1.6.1, and one of the new extensions that work with this version is one called ConfirmEdit. It can be configured to challenge “users” with a captcha upon account creation, as well as on each page edit (even only on page edits containing a URL).

For now, I’ve set the wiki up so that all edits require an account (no anonymous edits - I can live with that), and all account creations must provide a correct simple captcha response. Once you’ve created an account, you should never see the captcha again.

I’m settling for the “SimpleCaptcha” implementation - it’s a math question rather than a nasty wordimage. As much as I hate captchas is general, I figure it’s easy enough to fire up Calculator.app when you create an account, and shouldn’t block anyone with visual challenges.

wiki.ucalgary.ca has been under a sustained spam attack all day. What started out as a minor irritation has grown into something that is impossible to ignore. The spammer is somehow getting around both Bad Behavior and Spam Blacklist extensions (I’ve blacklisted their URLs, but they keep getting edits into the system). This is one of the more frustrating aspects of trying to do things in an open manner. If there is the slightest possibility that something will be subverted for spamilicious purposes, it will be. And most likely it will happen before more than a handful of legitimate users are able to take advantage of a service.

These cretins are being rather clever (or, they’ve gotten some good Script Kiddie l337 tools) because they’re coming from many different (and changing) IP addresses, and each edit is accompanied by its very own account creation. So I can’t just block IPs, or roll back all edits by a user. So, I’ve had to disable account creation for now until I can figure out wtf to do about this.

To the spammer(s): may you rot in the most insidious inner circle of hell, reserved for parasites like yourself who find it necessary to suck energy and resources from (otherwise) free and open educational resources.

wiki.ucalgary.ca got hammered by a vindictive wiki spammer last night. But, here’s the thing - the spam prevention blacklist worked perfectly. The spammer wasn’t able to add any of their own links to the wiki. So, they decided to punish me by vandalizing 50 of the most popular pages on the wiki with an apparently random (and invalid) spam URL.

The software they used to do this evil deed automatically created a new account for each edit, and the whole thing took them less than 10 minutes to do. It took me 45 minutes to undo, even with rollbacks etc… because of their insidious creation of 50 separate accounts for 50 separate edits. I would have just reverted back to a nightly database backup to blow them all away in one fell swoop, but we had actual valid users making actual valid edits, and I won’t blow any of that away. Better to manually remove the detritus than to lose a single valid edit.

I’ll be installing Bad Behavior today, when I get a chance It’s not like I have anything better to do than to play a game of Wiki Detente with a cretin who would vandalize an open academic resource because I wouldn’t let them add their link to their ViagraCasinoPenisEnlargement Google Juicer website factory…

The signature used by this roach shows up on a few sites on a quick Google. This is insane.

Update: I just installed Bad Behavior for MediaWiki - took a whopping 60 seconds to install and configure. I’d tried a previous version, but it got a bit, well, overeager about blocking stuff. To the point that even I couldn’t view or edit anything. Had to kill it last time. Hopefully this time will be better…

Akismet is the “official” WordPress response to the soul-sucking rampages of blog comment spam. It promises to make spam magically vanish by harnessing the Hive Mind to banish spam en masse. But it doesn’t work. I’ve been getting a fair amount of spam approved by Akismet as ham, when they are obviously spam. Not sure what’s going on there, but I’d guess that since anyone can flag comments as spam/ham, that the spammers are getting in the game themselves. Total guess though.

A couple of weeks ago, I turned off Spam Karma 2 to see how Akismet performed now that the system has had a few months to “warm up”. The result wasn’t exactly impressive. False negatives, false positives, and excessive moderation.

I could live with a few false negatives - the occasional spam slipping through the cracks and appearing on my blog isn’t the end of the world. But I’ve also had a couple of false positives. Valid comments banished by Akismet. I can manually resurrect them, but what If I don’t check regularly? It’d be really easy for false positives to get lost in the sea of spam (ick).

Also, Akismet routinely pushes comments into moderation purgatory. Someone attempts to post a valid comment, to be rewarded with an “I don’t trust you. Please wait for your comment to be blessed by the High Priestesses of Blog before being deemed worthy of being displayed here.” OK, it’s not exactly as rude as that, but the sentiment is the same, and not exactly conducive to conversation.

So, I’m going back to Spam Karma 2. It rocks hard, and is intelligent enough to block spam and approve ham without intervention. It even has an Akismet plugin for SK2 to let me harness the Hive as a last resort. But even that limited role of Akismet has proven to be the only weak link in SK2’s otherwise impervious armour, so I’ve downgraded Akismet’s influence from “normal” to “moderate.”

Can’t say enough good things about SK2. Since I first started using it, back when the world was young and Grandfather Bear roamed the forest, SK2 has nuked over 8700 spam attempts. About 100 attempts per day, and for 99.99% of them, I don’t even get notified. And so far there have been zero false positives (it keeps the comments and I periodically eyeball it to make sure).

I’d installed Spam_blacklist back in October, but we’d been getting the occasional spam attack since then, lately every single weekend. So I just dug into the Mediawiki config, and realized that Spam_blacklist never got properly configured on my server, meaning it was essentially running wide open. Crap. What a waste of time that was. Paul and I have removed dozens/hundreds of spams over the last few weeks, and I was assuming Spam_blacklist was enabled properly. Oops.

So, if you’re running Mediawiki, be sure to carefully (re)follow the instructions, especially Morbus Iff’s patch for 1.5 compatibility (I had to manually patch the file, since patch was barfing on an unmatched }).

The wiki is now configured to pull the blacklist from meta.wikimedia.org every hour, and will check edits against both that blacklist and the one we’re growing ourselves.

Things to watch out for:

• If you’re running the latest MediaWiki (1.5.x), be sure you’ve applied Morbus Iff’s patch.
• Also, be sure you’ve moved your local Spam_blacklist page to Mediawiki:Spam_blacklist so the plugin can see it. It apparently must be in the Mediawiki: namespace.
• Fix your copy of load_lists.sh to use the proper URL for updating the blacklist - bots have to use http://meta.wikimedia.org/w/index.php?title=Spam_blacklist&action=raw
• If you want, add the load_lists.sh file to your crontab so it runs automatically. I think the plugin tries to do that, but better safe than sorry
• Test your wiki’s now-hopefully-working antispam measures. Try to edit a page with a URL from the shared blacklist, then one from your own. Hopefully both attempts fail.
• (hopefully) relax a bit, and with luck the spammers will move on to greener pastures…

Why on earth isn’t this plugin included with the core Mediawiki distro? And why hasn’t the download been updated for 1.5 compatibility yet? It would be great to keep it up to date, and to make it easier to get it running properly. There are likely a LOT of Mediawiki instances that aren’t configured properly, making all Mediawiki instances nice, juicy targets.

Over the weekend, wiki.ucalgary.ca got hammered by a(n apparently) coordinated and distributed spam barrage. Hundreds of pages hit, new pages created, talk namespaces crapped into, etc…

I think I saw part of it happen in “real time” - I was watching a movie with Evan, with my Powerbook plugged into our TV (the only DVD player in the house), and every now and then I heard the system beep. After the movie ended, I saw the Watchmouse monitoring page for wiki.ucalgary.ca saying there was trouble connecting, and the main wiki.ucalgary.ca page was showing a MySQL connection error. Reloading the page made it go away, so I didn’t pay much attention.

Paul and I just spent the better part of an hour going through the wiki and delousing it, and I sure hope we didn’t inadvertently nuke a “real” edit, or spank a “real” user. The last thing I want is collateral damage in this silly battle against spammers.

I just can’t put into words how frustrating this is - we run a service for the use of the University community, to enhance the practice of teaching and learning online - and spammers run repeated drive-by-shootings spraying their crap all over it. It’s not just simple vandalism, though. It diminishes trust in the resource - why would a prof put stuff in it, if they can’t protect it from spam?

There has got to be a better solution to protecting a wiki from spam. I’ve got the Spam Blacklist extension for Mediawiki installed (and keep adding my own regex patterns, and periodically grabbing the blacklist from meta.wikimedia.org) - but that only helps protect against known spammers. I’ll be adding Bad Behavior (thanks to Paul for the link) to see if that helps.

Is there a tool available to go into a wiki and nuke any spam it finds? If spam keeps getting through the filters, there should be a way to yank it out again…

The Wiki Spammers are digging in. It looks like they’re falling back to brute force manual labour to spread their wares. The U of C Wiki was just hit by a moron who took 20 minutes to make 114 edits. It took me about 20 minutes to undo all of this clown’s links to whatever it was he’s trying to link to (using CSS to hide the links).

Then, at the last minute, I discovered the easy trick to nuke all contributions by a user. There is a handy page listing all of this roach’s contributions, and Sysops can easily nuke each and every one of them from this page, in no more than a minute or two.

So, not only is your URL thrown into the blacklist, but I can easily nuke any crap the gets through the filter. Begone, Mama71! Crawl back under your rock, and stop vandalizing an open educational resource. Surely there is something better you could be doing with your time.

Bring. It. On.

Update: Aron was apparently hanging out in the #mediawiki IRC channel when this was going on, and it looks like it was a coordinated attack felt across the Mediawikisphere. Man, that sucks. Mama71 just burned off a lot of Whuffie, vandalizing shared and open academic resources around the world. I’m guessing Mama71’s next incarnation will be as a roach, or lower. Not fun

I switched my antispam tool from Spam Karma 2 to Akismet last week - wanting to play with Matt’s new toy, and see how a distributed spam blocker would work out.

Generally, it was pretty good. But, I had to moderate several comments. I’ve almost never had to do that under SK2 - it just works, totally, and invisibly.

But, I just checked my WordPress dashboard, and saw an interesting item: Owen Winkler: Spam Karma 2 Akismet Plugin BETA. Hmmm… Akismet as a plugin for SK2? Cool!

Turns out, SK2 will do its usual magic, and if it’s still unsure about a comment, it will then feed it to Akismet for clarification. The best of both worlds! Very very cool.

I just installed Akismet - the “official” antispam solution for WordPress. Ok. It’s not really official, but it’s written by PhotoMatt - the lead of WordPress - which makes it official enough.

I’m a bit nervous about deactivating Spam Karma 2 - which has performed absolutely flawlessly (and silently), but am curious about the distributed spamblocking approach used by Akismet, as opposed to a personal blacklist used by SK2.

And, if Akismet fails miserably, it’s just a matter of clicking two links in the Plugin Manager to go back to SK2. One caveat to Akismet: you need a Wordpress.com API key to activate it. Download a copy of Flock to get one if you don’t have one already.

I just poked around the Spam Karma 2 reports, and realized that for the first time, ever, it’s showing more actual approved comments than denied spam attempts. Usually, there are hundreds of spam attempts listed for every successful comment - I never see the spam attempts, except for in these reports.

But, here’s what Spam Karma 2 is telling me:

There were 33 successful comments since I last looked, and only 31 spam attempts. Normally, there would be about 100-200 spam attempts listed here. Of course, none of them would actually make it to the blog, and I wouldn’t even know about them unless I bothered to check the spam log…

Could it be that the tide of spam is finally turning? Or is it just a slow day for the spammers? Maybe the Nigeria+Microsoft partnership is starting to bear fruit?

Update: Well… That didn’t last long. The spammers appear to be back in full force. I guess they just took the weekend off. Here’s the Spam Karma 2 info from this morning:

At least I don’t have to see or deal with any of the spam

They sure are persistent little buggers. I think I’ve reverted about 50 pages of wiki spam in the last week, on 2 wikis. The little cretins just won’t take a clue. They’re just smart enough to be able to switch or spoof IP addresses to get around the blocks and bans, but not quite smart enough to realize that I won’t let them win.

And, looking at my Referrer Karma blacklist, there are spamroach URLs in there that make even me blush. I mean, these people have nothing better to do than to register dozens/hundreds of obsene URLs, then feed them into their little script kiddie auto-blog-spammer software and see if they can scrape some Google Juice out of it.

Sorry, punks. Not on my watch.

I do get the sinking feeling that the wikis that I babysit will eventually act as some form of “deadman’s switch” - I picture myself in my 90’s, on my deathbed, worried about the spam that is about to infiltrate the online resources that I’d been stewarding…

There has got to be a better solution to the wiki spamroach problem than manual labour - that doesn’t scale, and the spamroaches know this. They are armed with better weapons, and are only refining their tools and techniques. We just keep plugging more fingers into more holes, hoping to stop the coming flood of spam (ew - that’s a nasty mental image… a river of that icky gelatinous goo…).

WordPress seems to have the spam problem nailed down pretty solidly - I have to exert almost no effort to keep spam from my blog. The little effort I do direct at it is in approving the occasional comment that gets popped into moderation rather than correctly passing or failing the spam tests. Why can’t wiki software be that intelligent about nuking spam?

Update: a quick google turned up these links:

• MediaWiki Anti Spam Features
• SpamBlacklist extension for MediaWiki

Update 2: Installed the SpamBlacklist extension for MediaWiki. They really left the process scary for non-geeks. Having to download the files individually from the webcvs repository, rather than just providing a nice .zip download, and not having the files configured to work “out of the box”. Regardless, it was a simple installation, and it’s now (hopefully) stopping spammers dead in their tracks. Or, at least slowing them down enough to piss them off enough to move somewhere else.

Spam Karma 2 just keeps on chugging away, protecting my blog from the scum sucking spam roaches of the world. The roaches are getting marginally more intelligent - starting to try to game the spam blockers.

In the wee hours of the morning, some spammer from somewhere in Asia tried to get onto some kind of whitelist by posting a couple of innocuous comments - with no bad links or scary words. Those 2 comments got through, and then they immediately tried to dump spam into the blog. Those comments were automatically killed by Spam Karma 2. It was able to make a distinction from harmless (although pointless) comments from link spam-infested roach fodder.

All I had to do this morning was manually moderate the first two whitelist attempt comments as spam, which took a grand total of 5 seconds. I love Spam Karma 2!