I just got this spam on my blog - it got through Akismet, as so many spams do lately, but it’s worth posting (at least in image form so the spammer doesn’t get any juice from it):

And, yeah. I hit the “Spam” button to file this in /dev/null even if the guy was honest(ish). I’m assuming he just ripped the text out of Wikipedia or something, and used it without proper linktribution…

My blog has been receiving spam in what looks to be a new wave of spam attacks. First, the spammers seed the whitelist by posting apparently innocuous comments with no URLs, or with a URL that doesn’t contain spam. Then, once they’re in, they wait a bit and then throw the switch. The spam starts a’comin’ and it sneaks through Spam Karma 2. Very annoying.

One thing I really like about SK2 is that it is standalone - it doesn’t rely on any network connection or other systems to flag stuff as spam. It just tracks IP addresses, user agents, and sniffs the content and URL for attempted comments.

But, that might be its weak point as well - by not harnessing the power of The Cloud, it’s more vulnerable to these kinds of guerrilla spam insurgencies. Once someone using Akismet has flagged someone as an evil spammer, everyone automattically benefits from that, without having to each individually flag the spammass as a jerkwad.

SK2 has served me well for quite some time. Here’s the current stats report:

Over 200,000 spams dealt with. But the number of moderations required is getting inconvenient - not impossible, but it’s becoming something I need to manage rather than just fire-and-forget, the way things used to be.

Now, with Akismet enabled instead, I’m at the mercy of The Cloud, but that might not be a bad thing…

After switching from BadBehavior+Spam.module back to Akismet, I assumed I’d be in for a bit of an onslaught of spam. I was braced for impact. I can’t believe the sheer volume of sustained attempted spam comments that are constantly being flung against this blog, 24/7 now. It’s peaked at several attempts per second, which was adding a bit of a load to the server as it struggled to thwart the forces of evil.

Shortly after switching to Akismet, and enabling the experimental spam detection, I was seeing this:

Now, that might not look like much, but it suggests that Akismet was having to reject attempts several times per minute. Fast forward 24 hours, and I see this:

Again, not looking like much, but the interval between Akismet interventions is getting longer. Either the spammers are slowly starting to give up, or this is just a natural lull. I mean, there can be several minutes now without an attempted spamment posting. Entire minutes!

Now, the downside of Akismet is that I can’t use it on any of my campus projects. The cost of licensing Akismet for the number of sites we have would be prohibitive, given our budget asymptotically approaching zero dollars (CDN).

OK. Even I am getting sick of the incessant "spam blocking update" posts, but I figure if it helps even one other person put the brakes on the attempts of the evil spamroaches, it's worth it.

So, here's the latest. I got frustrated with the number of spamments that snuck through the combo of Bad Behavior and Spam.module, so I disabled both. I've reverted to using only Akismet.module, with the experimental spambot detection/prevention enabled.

And, so far, it's doing a better job at blocking the roaches. I've got no idea if it's also blocking legitimate hu-mans, though.

One nice thing about Akismet.module vs. spam.module - with Akismet's experimental spambot prevention, it's closer to acting like Spam Karma 2, where if you smell like a roach, you don't even get close enough to pop the lid off your can of spray paint.

I'll have to look into updating Akismet.module for Drupal 5. There's really no sense in actually moving to D5 without spam blocking. That'd be kind of silly.

As an aside, I was looking through some of the logs, and found an interesting user agent, which led me to the product website for one of the evil spam roach comment bot factory applications. They have disclaimers on the site saying they don't condone using their product without the permission of the blog owners. What? Permission? What a frakking load of ass-covering crap that is. Yeah. You're going to give someone permission to aim a program titled "Blog Post Uzi" - because, you know, Uzis are all warm and fuzzy, and the kind of thing that friends give permission to other friends to point at each other. Yeah. Permission to spray the output of a concealable assault gun. Whatever. Karma's going to catch up to you in spades, my friends at Promo Arsenal (dot com).

I was just clearing out the last of the comment spam - I'd let it stew in the database (unpublished) but thought I'd take a quick peek to see if there were any false positives. I thought I'd found one - a comment marked as spam, but with content portraying sympathy with my plight against these spammers - that they must be stopped.

A Spammer Responds (screenshot): This spam roach was trying to get whitelisted by commiserating on the evils of spammers… It didn't work - Akismet sniffed it a mile away.

So, without thinking, I clicked "submit ham" - to tell Akismet that it was a bona fide comment. Then, I checked the URL to see which friendly blogger was commenting.

And got a spam site.

The spammer was trying to get through the filters by reading my recent post and trying to get whitelisted by posting something not spamish at first glance. But, Akismet had stopped him in his tracks - until I clumsily intervened. It's now re-flagged as Spam, and banished to that special inner circle of hell reserved for these roaches.

The spammers started trailing off not long after I wrote the previous post - before hitting their target of 20,000 spam attempts in 24 hours. They punked out at about 18,000 - then I closed the door with the Bad Behavior module.

It was kind of interesting leaving the spammers swarming around my blog as a honeypot, but the load was just getting annoying. Since enabling Bad Behavior, Akismet has had to deal with less than a dozen spammers getting through in about 24 hours - and I haven’t had to deal with (or even be aware of) any of them. That’s a wee bit of a change…

Bad Behavior makes me a bit nervous though, because it is rather unforgiving by design. If it thinks you’re a spammer, or if your IP has been used by a spammer, you’re locked out. No second chances. That’s good, but it’s also a bit authoritarian. There’s also no admin interface for it, so if I want to unblock someone, I have to dig around in the database to nuke the appropriate records.

I’ll keep an eye on things, but it’s pretty cool knowing that this blog could handle a pretty intense load without breaking a sweat, that spammers will not be getting in, and that it takes basically no effort on my part to maintain things. Very cool.

The onslaught just keeps coming. It’s on pace to easily meet 20,000 spam comments in 24 hours. I had a very small handful of false negatives, but they were easily dispatched by clicking a couple of checkboxes on an admin page.

20,000 spam attempts. Peaking at multiple attempts per second, with over 500 spam bots simultaneously spidering/spamming the site. Drupal seems to be holding its own, triggering throttles to shut down higher-load functions so the site remains responsive.

And Akismet is dutifully blocking the roaches from getting through. Their success rate just keeps dropping - down to 0.1% success rate, and even that is only temporary, until I manually remove the small handful that gets by Akismet’s watchful gaze. The spammer’s effective success rate is exactly 0% - not asymptotally approaching 0, but exactly 0. Zero. Absolute Zero. Cold heat death for spam.

I’m pretty sure they’re sticking around here because the bots think they are succeeding - Drupal accepts the comment, so the bots think it worked. What they don’t realize is that the comment is immediately unpublished by the Akismet module - as soon as the Akismet retuns its spam flag.

Thankfully, Dreamhost appears to be handling it like a champ. Not even breaking a sweat. I’m pretty sure that if this blog was still back on GoDaddy’s servers, it would have curled up into the fetal position and mumbled quietly to itself…

I'd tried Akismet before, and wound up reverting to Spam Karma 2 - actually, I think SK2 was interfering with Akismet, so that likely wasn't a fair comparison, since both were running at the same time.

But, I've been running the Drupal Akismet module for 10 days now, and it's been performing absolutely perfectly. For example, this blog has been under a sustained spam attack for the last 12 hours or so - over 400 600 spam attempts just last night (200 just while writing this post) - and not a single one of the roaches got through. I just went through the Akismet comment moderation queue to look for false positives, and there wasn't a single one. So it's batting 1000 under a significant spam attack.

Mad kudos to Akismet, and to Markus for porting it to Drupal!

Actually, I'm pretty impressed at how this blog is running under this spam attack - it's still responsive, pages load quickly, and posting new content and comments is still working. Drupal's handling the extra load without breaking a sweat.

I'lll be keeping the Akismet module running on my blog, but will keep playing with the Spam.module update on our campus server due to licensing requirements for Akismet. Our use falls under the "commercial" category, and with the number of Drupal sites we're using, the cost can't be justified (yet - maybe if we get hit by spammers that don't get blocked by spam.module we'll adjust things to find the cash).

Update: Over 2000 attempted spam comments in the last 24 hours, and every single one was stopped by Akismet. Here's a screenshot from the Akismet moderation log - I didn't have to see any of these, and they were coming at me every in bursts of up to 1 spam per second for 24 hours…

Akismet Spam Attack: A screenshot of the Akismet Drupal module moderation log, during a sustained spam attack where Akismet blocked 100% of attempted spams, with no false positives.

Update 2: It's been almost exactly 24 hours since I first wrote this entry. Since then, an additional 2000 comment spam attempts have been successfullly blocked by Akismet. And a whopping 8 spam comments got through to my blog. 8. That's it. Out of over 3000 attempts in a day and a half. That's roughly a 0.267% success rate for the spammers. But, the economics of it make even THAT a worthwhile use of their time.

I debated doing something more proactive to stop the spammers altogether, but then thought that it's probably better for them to leave their bots pointed here, getting no benefit at all, than randomly spraying their spam across the 'net and maybe hitting someone's blog that isn't using an effective spam blocker. The way the Drupal Akismet module works, the spammers think they're getting every comment posted here, but the module immediately unpublishes their spam as soon as Akismet responds. That's a pretty sane way to set up the block - don't tell the spammers that they're wasting their time, just nuke their spam without a whimper…

Akismet is the “official” WordPress response to the soul-sucking rampages of blog comment spam. It promises to make spam magically vanish by harnessing the Hive Mind to banish spam en masse. But it doesn’t work. I’ve been getting a fair amount of spam approved by Akismet as ham, when they are obviously spam. Not sure what’s going on there, but I’d guess that since anyone can flag comments as spam/ham, that the spammers are getting in the game themselves. Total guess though.

A couple of weeks ago, I turned off Spam Karma 2 to see how Akismet performed now that the system has had a few months to “warm up”. The result wasn’t exactly impressive. False negatives, false positives, and excessive moderation.

I could live with a few false negatives - the occasional spam slipping through the cracks and appearing on my blog isn’t the end of the world. But I’ve also had a couple of false positives. Valid comments banished by Akismet. I can manually resurrect them, but what If I don’t check regularly? It’d be really easy for false positives to get lost in the sea of spam (ick).

Also, Akismet routinely pushes comments into moderation purgatory. Someone attempts to post a valid comment, to be rewarded with an “I don’t trust you. Please wait for your comment to be blessed by the High Priestesses of Blog before being deemed worthy of being displayed here.” OK, it’s not exactly as rude as that, but the sentiment is the same, and not exactly conducive to conversation.

So, I’m going back to Spam Karma 2. It rocks hard, and is intelligent enough to block spam and approve ham without intervention. It even has an Akismet plugin for SK2 to let me harness the Hive as a last resort. But even that limited role of Akismet has proven to be the only weak link in SK2’s otherwise impervious armour, so I’ve downgraded Akismet’s influence from “normal” to “moderate.”

Can’t say enough good things about SK2. Since I first started using it, back when the world was young and Grandfather Bear roamed the forest, SK2 has nuked over 8700 spam attempts. About 100 attempts per day, and for 99.99% of them, I don’t even get notified. And so far there have been zero false positives (it keeps the comments and I periodically eyeball it to make sure).