Requests for Anti-Spam Features [Archive]

Nick

09-25-2009, 03:52 AM

Let's build a wishlist of all the anti-spam options you'd like to see in Hotaru (for social bookmarking). I know Akismet and other automated options work well on blogs, but bookmarking sites attract much more spam because they are essentially giving away "free backlinks". That's why we need a number of manual anti-spam tools as well.

I'll start. Please add more if you can think of them.

Registration

- Captcha done
- Email confirmation done
- Block by IP done
- Block by email type done
- Block by username done
- New user notification with option to approve/deny done
- Hard/soft user deletion done
- Plugin for StopForumSpam.com (http://www.stopforumspam.com/) (suggested by JonH below) done

General

- Add users to a watchlist done - by giving them undermod status
- Edit user permissions to submit, vote, comment, etc. done*

* only useful if plugins add their own permissions

Posts/Comments

- Block posts from being re-submitted done
- Block posts by domain done
- Daily submission/comment limit done
- Restrict submission rate (e.g. 10 mins between posts) done
- Moderate first X submissions/comments done
- Moderate all submissions/comments from watchlist users done
- Enable/disable comments sitewide and on individual threads done
- Close threads, but retain existing comments done
- Limit urls allowed in posts/comments done
- Hard/soft delete posts/comments done
- Plugin for Akismet (http://akismet.com/) done
- Flag posts/comments done for posts
- Report posts/comments (by email)

Other

- allow users to filter what they see by user, domain, category, friends

angolanmade

09-25-2009, 05:43 PM

Awesome list, getting a strangle hold of spam should be a priority, specially for a social bookmarking site!

runnertalk

09-25-2009, 06:57 PM

Awesome list, getting a strangle hold of spam should be a priority, specially for a social bookmarking site!

Agree, on my site there are more spam users than it is real users so this is a very important part of the cms.

JonH

09-26-2009, 07:03 AM

I've always thought a centralized list or database somewhere where all Hotaru sites can 'report spam' would be nice.

It would be like a shared Pligg antispam.txt on hosted site somewhere that a plugin can read or write the emails, usernames, websites, ip addresses etc of spammers. If you ever google a spammers name sure enough they've hit all the social bookmarking sites on Google.

After a spammer's info is on there a certain length of time it would automatically be deleted to keep the list from being to big and slowing everything down.

Oh I forgot about this site...
http://www.stopforumspam.com/ has an API - might be able to use them!

Nick

09-26-2009, 08:21 AM

Oh I forgot about this site...
http://www.stopforumspam.com/ has an API - might be able to use them!

Nice find! Looks like a great solution to your "centralized list" suggestion.

davidshq

10-09-2009, 02:19 AM

- I really like the approve/deny feature for new users. This could be exceptionally helpful.
- The daily submit/comment limit is an absolute must.
- The ability to review first submission/comment is another must.
- Akismet is one of those must have features for me. Done right it can take care of 99% of spam on its own.
Okay, I think thats it for now. :)

Nick

11-08-2009, 03:08 PM

I'm making a plugin for StopForumSpam (http://www.stopforumspam.com/) and it's going to be kick-ass awesome at blocking spam registrations!

Imagine if a spammer joins my Hotaru site and I killspam or delete him, that information will be added to the SFS database so if he then tries to register at your Hotaru site, he'll go straight into moderation before he can do any damage. The combined effect of dozens of Hotaru sites automatically adding their spam users to the SFS database will be very interesting to see.

The biggest challenge will be keeping us from accidentally adding ourselves or our honest users to the database! :eek:

Update: It's half done. It blocks users okay, but I still need to make a settings page and a function to add bad users to the SFS database.

Update 2: Done!

Nick

11-11-2009, 04:31 PM

A little bump here to draw attention to the first post. Nearly finished everything!

JonH

11-12-2009, 02:14 AM

I'm making a plugin for StopForumSpam (http://www.stopforumspam.com/) and it's going to be kick-ass awesome at blocking spam registrations!

Update 2: Done!

That's really slick. Other CMS's have plugins for this like SMF so the spammer database is already quite large. It's 1 central shared database right?

Nick

11-12-2009, 03:06 AM

Yep, with nearly half a million spammers in it!

When killspamming or deleting a user, I made it so you have to tick a checkbox if you want them to be added to the SFS database, otherwise you might accidentally add any test users you're playing with.

I also put in another checkbox to let you add them to Hotaru's own blacklist. Your Hotaru "blocked list" doesn't let people register at all, whereas the StopSpam plugin will let them register, but give them pending status if they are spammers (just in case). The blocked list is always checked first, so if the same spammers keep hitting your site, you can permanently shut them out without having to check the SFS database each time and then deleting them them again and again.

Saying that, we've also got the killspam vs delete difference. Killspamming someone leaves them in the users table so they couldn't register with the same name/email again anyway, whether they are on the Blocked List or not. Deleting them will of course remove them completely.

This is seriously spam busting overkill. Short of hacking the site, I can't see any way a spammer could get in and actually make a post or comment. If by some miraculous Houdini magic they did get to post something, and Akismet didn't catch it, you can set up Hotaru to send you or your moderators an instant email about new posts or comments, or let other users bury it into deletion.

If you're unsure if a new user is a spammer or not, just give them undermod status until you can be sure.

Spam-free social bookmarking is what Hotaru was built for! :D

rushnp774

11-26-2009, 09:36 PM

Fantastic list so far, and I'm glad anti-spam features are so high up on your priority list! I don't really have much to add, but I do have a question.

When an IP or domain is added to a global block list, what if there are some legitimate users from that domain that you'd like to let register? i.e. Not everyone from @gmail.com is a spammer. I'm sure the devs of those kinds of modules have thought about things like that, but I don't know. Maybe if an adress from that blacklisted domain tries to register/post, the first one or two would go to a moderator's queue first, when they're approved they'd go to a whitelist? Just thinking out loud here, and I'm sure this idea has already been thought of.

Nick

11-26-2009, 10:35 PM

Hotaru never adds anyone to a block list automatically.

You can choose to add someone to the StopForumSpam database. If a different person with the same IP tries to register, he will go into moderation and can be approved by the site admin, unless the admin has chosen to block "flagged" people immediately.

Hotaru's own blocked list is managed entirely by the site admin. When killspamming or deleting a user, you can choose to add that person's username and email address to the blocked list. If you want to add their IP address, you'll have to do it manually, exactly because of the false positive problems that could arise.

rushnp774

11-27-2009, 12:28 AM

Makes sense. To be honest, I don't know much about anti-spam features, nor the coding behind the scenes. The only real anti-spam experience I have is that I use Akismet on my Wordpress blogs, and that does 99.9% of the work for me.

I did play around with Drigg though, and noticed that there was a global blacklist of IPs and domain names that the admin could add to. I always thought that was a great idea since most of the spam in my blogs comes from addresses that end in ".ru", and adding the ability to flag/block users from those domains would be quite helpful. Although, the central database (similar to Akismet or the other one mentioned in this thread) is a fantastic idea that should be looked into strongly.

I'll play around with the core anti-spam features as well as the included modules to see if there's anything I can think of adding that would help improve Hotaru's anti-spam capabilities.

carlo75

11-27-2009, 06:40 AM

Killspam and deleting differences are really importants, I don't understand why anybody make that before in pligg -.-

What do you think on a Block by extension in posting?
I've made something for pligg, for stop submitting .ru and other undesired domain extension...

Nick

11-27-2009, 08:55 AM

What do you think on a Block by extension in posting?

If you go to your Blocked List in Admin and add .ru as a URL, then all you would need to do is add an extra condition to the checkBlocked function in submit.php:

public function checkBlocked($url)
{
// Is url blocked?
if ($this->isBlocked('url', $url)) {
return true;
}

// Is domain blocked?
$domain = get_domain($url); // returns the domain including http
if ($this->isBlocked('url', $domain)) {
return true;
}

// Is domain extension blocked?
$host = parse_url($url, PHP_URL_HOST); // returns www.google.com
$ext = SOMETHING HERE TO GET THE .COM;
if ($this->isBlocked('url', $ext)) {
return true;
}

return false; // not blocked
}

So, if someone can finish that last part, then we can block by domain extension! :D

carlo75

11-27-2009, 09:22 AM

$ext = substr(strrchr($host, '.'), 1);
if ($ext == 'ru') { //Add more extensions here
echo "Domain Extension not Allowed";
return true;
} else {
return false;
}

something like that?

Nick

11-27-2009, 09:26 AM

Maybe this?

$ext = substr(strrchr($host, '.'), 1);
if ($this->isBlocked('url', $ext)) {
return true;
}

carlo75

11-27-2009, 09:28 AM

probably so better :p

Nick

11-27-2009, 09:41 AM

You might need to include the dot or just let admins put ru in the Blocked List instead of .ru

If you try it and it works, let me know and I'll put it in the plugin. Thanks! Team effort, high five! :cool:

carlo75

11-27-2009, 09:55 AM

With ru the extension domain is correctly blocked

With .ru doesn't work...

Tried with www.google.ru

Nick

11-27-2009, 10:19 AM

So from a user's point of view, is it better to put ru or .ru in the blocked list?

Option 1:

User adds ru to the blocked list.

// Is domain extension blocked?
$host = parse_url($url, PHP_URL_HOST); // returns www.google.com
$ext = substr(strrchr($host, '.'), 1);
if ($this->isBlocked('url', $ext)) {
return true;
}
Option 2:

User adds .ru to the blocked list.

// Is domain extension blocked?
$host = parse_url($url, PHP_URL_HOST); // returns www.google.com
$ext = substr(strrchr($host, '.'), 1);
if ($this->isBlocked('url', '.' . $ext)) { // dot added here
return true;
}
I prefer option 2 myself.

carlo75

11-27-2009, 10:34 AM

Option 2 sounds good for me too...
so passed!

Nick

11-27-2009, 10:51 AM

so passed!

Yep, it will be in Submit 1.7. Thanks for the idea and helping to implement it.

rushnp774

11-27-2009, 07:55 PM

Maybe this?

$ext = substr(strrchr($host, '.'), 1);
if ($this->isBlocked('url', $ext)) {
return true;
}

With ru the extension domain is correctly blocked

With .ru doesn't work...

Tried with www.google.ru

I don't know the exact string concatenation commands in PHP, but going from what Carlo said (it works without the period), could you change the line in your PHP code to something like:

if ($this->isBlocked('url', "add a period here" + concatenation operator + $ext))

If I'm understanding the line of code correctly, it calls the isBlocked function (a custom function you made?) with the parameter "url" (I'm guessing this came from the parse_url) along with an extension. I'm still not seeing anywhere that a period is inserted into the string.

Just trying to dive headlong into this and wrap my head around this after doing a bit of reading, but I'm sure I'm off base a bit. If so, just ignore what I have to say :D

Nick

11-27-2009, 09:33 PM

I'm still not seeing anywhere that a period is inserted into the string.

It's there, up in Option 2 (http://hotarucms.org/showthread.php?125-Requests-for-Anti-Spam-Features&p=907&viewfull=1#post907) ;)

if ($this->isBlocked('url', '.' . $ext)) { // dot added here

rushnp774

11-27-2009, 09:38 PM

Bah, I need to learn how to read. Did carlo try the code mentioned in Option 1, which is why ".ru" didn't work for him?

smiera

08-14-2010, 07:05 AM

Great collection and solution for new comers like me.. thanks again.

richrf

08-15-2010, 03:15 AM

Hi,

I am new to the forum but am familiar with Pligg and other CMS packages, e.g. Wordpress and Joomla.

I was looking over your anti-spam features but couldn't determine whether there are plugins that might provide similar anti-bot protection as does the popular CMS plugin called Bad-Behavior (http://www.bad-behavior.ioerror.us/). Is there a similar plugin available for Hotaru? Thanks for the info.

Rich

Nick

08-15-2010, 03:35 AM

Hi Rich, no, there's no Bad Behavior plugin yet. Maybe someone will volunteer to make it?

richrf

08-15-2010, 02:48 PM

Thanks for the reply Nick. WP-SpamFree (http://www.polepositionmarketing.com/library/wp-spamfree/), another Wordpress anti-bot plugin is an improved version of Bad Behavior. I have used both on Wordpress blogs and they are absolutely invaluable. I think Pligg has a Bad Behavior plugin. If I had some idea of the cost of programming it I might opt to pay for development should I decide to use Hotaru.

Rich

petsagouris

08-15-2010, 03:15 PM

Thanks for the reply Nick. WP-SpamFree (http://www.polepositionmarketing.com/library/wp-spamfree/), another Wordpress anti-bot plugin is an improved version of Bad Behavior. [...]After some reasearch the BB author has some objections (http://www.bad-behavior.ioerror.us/category/wp-spamfree/).

richrf

08-15-2010, 04:57 PM

After some reasearch the BB author has some objections (http://www.bad-behavior.ioerror.us/category/wp-spamfree/).

Thanks for the reference. I think the author's comments are well worth considering. My own experiences with WP-SpamFree are very positive but, as the BB author notes, there are issues worth considering. I believe either approach is worthwhile. and in my case somewhat necessary, since my site is a frequent target for bot attacks.

petsagouris

08-15-2010, 05:00 PM

I would vote for a Bad-Behavior plugin, not a WP-SpamFree.
I haven't done any plugin for Hotaru yet. It is doomed to happen sooner or later, but it can't happen now.

richrf

08-15-2010, 05:23 PM

I would vote for a Bad-Behavior plugin, not a WP-SpamFree.
I haven't done any plugin for Hotaru yet. It is doomed to happen sooner or later, but it can't happen now.\

Either with probably be adequate. My site has lots of traffic including bot traffic. Both plugins were more than adequate with WP-SpamFree doing a slightly better job - with the provisos as noted in your reference.

kameleon

02-25-2012, 02:38 AM

Is it possible that you add and create 'Spam Trigger' plugin for hotaru like plugin used by Pligg:

Spam Trigger - Anti-Spam and Profanity Filter
http://forums.pligg.com/free-modules/20191-spam-trigger-anti-spam-profanity-filter.html

- which works in way that you have text file in which you enter list of spammy words, and when someone submits bookmark that contain any of words meantioned in that text file - that bookmarks either gets deleted or get some status and goes first to reviewing - so it doesn't show on home page instantly...

faria18000

06-12-2012, 09:19 PM

Hi, currently i am using pligg cms, but due to heavy spamming i am in fear about my google adsense. There is a spam trigger, but this spam trigger only can block english language sites. When an user submit casinos and gambling content from other languages then spam trigger can not find out it. So everyday i am getting a lots of spam submission and google is giving me warning continuously.

I thinks it is possible to check spam if i can block other language sites except English.

Is there any way to block all the languages except English? Then i will sure use hotarucms for my bookmarking site.

Hope i will get a solution. I have also posted a thread in pligg forum, you can check it below:

http://forums.pligg.com/questions-comments/22425-how-do-i-block-non-english-stories-submissions.html

frytec

06-15-2012, 05:51 PM