It's clear enough from the title. I ever talked this issue with RSnake but he seems to be not interested with this. But last night when I tried this DoS tehcnique on my very own blog,and the result my blog was suspended due to CPU Usage Exceeded within a few minutes.

Should we start to stop those sitemap generator to crawl our site?

Are you talking of tools like Google sitemap generator? Or standalone apps?

Online sitemap generator,ex : http://www.pctimelimit.com/siteindex.php.

And I wrote a small program to do the execution loop. It's still works last night (using the http://pctimelimit.com sitemap generator), but now it gives me a 502 error.

-Jackson

Am I the only one getting these errors?

Notice: Undefined index: url in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 346

Notice: Undefined index: url in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 347


Looks like someone forgot to turn error reporting to not display O.O

@CrYpTiC_MauleR : it works fine just now ..

Still errors for me, I'm getting 15 undefined index errors.

Notice: Undefined index: url in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 346

Notice: Undefined index: url in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 347

Notice: Undefined index: host in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 87

Notice: Undefined index: url in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 348

Notice: Undefined index: host in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 87

Notice: Undefined index: url in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 349

Notice: Undefined index: maxpages in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 351

Notice: Undefined index: www in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 355

Notice: Undefined index: www in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 356

Notice: Undefined index: index in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 359

Notice: Undefined index: index in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 360

Notice: Undefined index: ses in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 362

Notice: Undefined index: ext in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 369

Notice: Undefined index: host in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 87
Root location:
Notice: Undefined index: url in /home/pctimelimit/data/www/pctimelimit.com/siteindex.php on line 419

Execution time: 0.00 secs
URLs parsed: 0
URLs extracted: 0


Awesome AnDrEw - That's The Sound Of Your Brain Crackin'
http://www.awesomeandrew.net/

Weird, it was fine the other day. But my point is that if a malicious user combining several online sitemap generator site to crawl a single site that will be a serious problem.

While I will agree that targeting a website for crawling by sitemap generators may place stress on the server there are a number of factors which need to be taken into consideration, and which may limit the impact this potentially has on the server and its resources. The first aspect which should be taken into account is bandwith. If the website is hosted on a free service such as Angelfire it may be susceptible to downtime due to their hourly and monthly limits, but then again this could be done solely with a browser and F5 (as in refresh), or too many HTTP GET requests (we have all seen these programs, or created them at one point or another). Other websites with higher bandwith allotments may use only a small percentage of given resources when struck with multiple crawlers.
Another issue is the .htaccess configuration. A majority of these crawlers can be blocked with either a mod_rewrite rule, a "deny from" statement, or both. Obviously there will still be a large number of requests made if the client does not understand the fact it is unwanted, but with even the smallest amount of bandwith this would most likely not cause a problem. One could even go so far as to firewalling off the IP address to keep any of its requests from ever being seen by the website.
Don't get me wrong it would still be a pain in the ass to look at or first deal with, but it's nothing outside of the actions performed by many search engine crawlers (I'm looking at you searchme).


Awesome AnDrEw - That's The Sound Of Your Brain Crackin'
http://www.awesomeandrew.net/

Yeah, i hosted on hostmonster with 4,5Tb of bandwith available, but it's down too :P Maybe hostmosnter took precaution for this kind of attacks. Thanks Awesome AnDrEw for your explantation! ;)