Kaironews was shut down tonight by our ISP for very excessive use of CPU resources on the server. I believe the problem was caused by spammers and ill-behaved search engines (I found one such instance of the latter).
Since those of us who use the site don't want this to happen, I've
- Temporarily enabled the bad behaviour module. This blocks some spammers at the Drupal level; it may or may not help reduce CPU usage.
- Banned the offending IP range which occupied 11 of the top IP's in terms of page visits this month at Kairosnews and used .99 GB's of the 4.7 GB's used this month. (Sorry Platypus Matt. If that happens to be you using multiple computers at your insitution to see if anyone replied to your posts, you are now effectively blocked at the server level ;-) )
- Limited news aggregator and search access to logged in members.
- Enabled Drupal's caching and throttling mechanism for anonymous users. If you are logged in, it won't affect you. But if you visit Kairosnews without logging in, you may not see the most recent content as Drupal will pull some pages directly from its cache. This should help as these tools were designed specifically for drupal.org and other popular sites using Drupal to survive a Slashdotting.
bad behavior module
:-P
And what's this about using multiple computers? I thought it was expected that we'd install bots on all the computers to make it look like more people are reading our posts. Guess I missed that memo...Must've gotten lost in all this Treet in my inbox. Just please don't install a Treet blocker. It's nutritious! And only $1.49. And it prevants prostate cancer. I read that at Net Gold.
Check out Barton's gaming blog at Armchair Arcade.
moving from spam to treet
I'm interested in why you have moved from spam to treet. Is that higher quality crap?
-----
Charlie | cyberdash
Bad Behavior did not turn out well on my site
I run a Drupal site on a hosting system that gives me root access to the server. We had problems with the BB module rejecting legitimate access. One example was that I had to edit its exclusion whitelist to allow Shiira running the nightly build of WebKit to access the site. I also found cacheing to be a problem. Actually, Charlie mentioned it might be why folks were getting blank screens or other incorrect results on our site. I never was able to find exactly what was slowing things down.
My ISP said that CPU usage was not a problem and that the machine was not swapping too much. Then the problem went away. But I have cut back on the modules I have enabled. I would like to find a program similar to Drupal but designed more for sites that have access to the command line and can install their own software.
I think PHP/MySQL is just slow. Our main site uses mod_perl to build dynamic pages from text files. Our Drupal's main screen takes 30-40 seconds to load (with little content). Our main site (with more graphics and text) rarely takes 5 seconds. The server pulls files off the net at rates that make me think the problems are not bandwidth. And I can configure Makefiles on the server almost as fast as I can on my Desktop G5 which runs at 2.1Ghz. So I really think the problems are in Drupal.
-- rdr
Drupal, performance, etc.
Certainly, you might have to update a whitelist to allow legitimate access for BB. As I understand it, BB uses blacklists and runs on the premise that legitimate http requests will provide user agent information. If you have a legitimate request which is not following general guidelines for the user agent, BB will reject it. That's how it works.
As for performance, your problem sounds strange. We were running many, many Drupal sites on the same server at Purdue. We did hit a wall at one point because of my limited knowledge of mysql configuration. The default conf for mysql is designed to run on any machine--even a very old one--and thus is not nearly optimized enough for newer machines with lots of memory. I had to do some research and some testing to get it run more efficiently again. That's the fault of mysql, just my lack of knowledge with configuring the tool correctly.
30-40 seconds seems like a long time for a Drupal site to load in my experience. I would suspect it is something else. What I would suggest is checking to see if there are any configuration problems anywhere that might cause your server not run mysql in an optimal configuration. Or you problem could be with a contributed module. Drupal core tends to run very efficiently, but I have seen instances where contrib modules caused problems. For instance, prior to 4.7, very large sites that ran pathauto had problems.
-----
Charlie | cyberdash
Re: MySQL
We ran slow-request checks and other tests. The only slowness (but not too slow) was from accesses by Drupal. We do have 4G of RAM on the machine, so we cache most of the queries, and we do not allow external access to the database.
VM use runs only 15-19%. With a load average generally 0.02, 0.02, 0.00 I will try removing path_auto.
By the way, Bad_behavior blocks access on a Mac by Shiira, Real Mozilla (i.e. compiled for X11), Camino, and curl. Firefox works. These problems have been discussed in one of the Drupal forums.
--
rdr