Cenzic 232 Patent
Paid Advertising
sla.ckers.org is
ha.ckers sla.cking
Sla.ckers.org
Ways to improve page rank, or deceptively get more users to your websites or away from your competition. Where you can discuss SEO (search engine optimization) issues as it relates to computer security. 
Go to Topic: PreviousNext
Go to: Forum ListMessage ListNew TopicSearchLog In
Targeting Web Crawlers With PHP
Date: December 18, 2006 12:35AM

I'm not at home right now, and I haven't had a ton of time to come online lately (plus I'm busy playing with my Wii), but I was thinking of a suggestion on how S.E.O. could be achieved very cheaply by targeting the crawlers from a numerous amount of search engines using PHP. I'm not one for trying to increase the monetary value of my website, but I figured by simply filtering User-Agents for the word "bot", or listing a few known crawlers, and then having the page only display plaintext keywords it'd have a similar effect to those sites, which add large numbers of keywords in the same font color as the background (so it's not entirely visible) at the footer of the page. It'd probably work a lot better this way as actual viewers would see one set of content, and the bots would receive whatever content it is that you'd want your site to appear with via search results.


Awesome AnDrEw - That's The Sound Of Your Brain Crackin'
http://www.awesomeandrew.net/

Options: ReplyQuote
Re: Targeting Web Crawlers With PHP
Posted by: jungsonn
Date: December 18, 2006 10:49AM

Interesting stuff Andrew.

Little out context rant:
Actually i was poking around today with some old logs and new server logs, i did something very sneaky, i did let Google index my old blog for a couple of months, then i deleted the whole blog, but kept the subfolder and wrote a new one, it is pretty funny to see Google is in a haste to update it with the new articles cause i deleted many old one which we're indexed very nice. it is interested to see which IP's they use for it, (and how long the bot stays) today it sat around 1 hour on a few articles. so im collecting these logs now, and see what i can deduct from it.

Options: ReplyQuote
Re: Targeting Web Crawlers With PHP
Posted by: rsnake
Date: December 18, 2006 01:34PM

There are two problems with SEO cloaking. 1) Search engines tend to move around, changing their names, blah blah (that one is pretty easy to solve) and 2) your competitors spoof Google and see that you are cloaking and then report you which completely kills your cloaking.

Two _very_ different problems.

- RSnake
Gotta love it. http://ha.ckers.org

Options: ReplyQuote
Re: Targeting Web Crawlers With PHP
Posted by: digi7al64
Date: December 18, 2006 06:00PM

For your amusement here is a site i have watched over the last 4 years game Google.com over and over and over again with its bot detection routine

Cached Version:
http://209.85.165.104/search?q=cache:W8PACjnHfloJ:www.opals-opals.biz/+opals-opals.biz&hl=en&gl=au&ct=clnk&cd=1

Real Version:
http://www.opals-opals.biz/

The interesting thing about this site is that is has a lot of keyword related pages (designed only for seo) that have been indexed by Google.com yet you can't actually get to them from the site itself (with much success i might add).

Therefore the problem Google has and always will have is the fact that it doesn't use a secondary (randomly named) bot designed to detect gamers.

----------
'Just because you got the bacon, lettuce, and tomato don't mean I'm gonna give you my toast.'



Edited 1 time(s). Last edit at 12/20/2006 05:14PM by digi7al64.

Options: ReplyQuote
Re: Targeting Web Crawlers With PHP
Posted by: anty
Date: January 06, 2007 08:13AM

One popular method to decide if a visitor is a bot or a human is called "ip delivery". You decide which content you want to show based on a list of ip-addresses. I personally use this method and haven't any problems with it. A very good source to get ips of popular search engines is http://iplists.com/ witch is updated on a regular basis.

anty.at

Options: ReplyQuote
Re: Targeting Web Crawlers With PHP
Posted by: rsnake
Date: January 09, 2007 04:01PM

Yah, Dan's lists are a good resource. I've actually been compiling a far more exhaustive list for the last several months, but it would take several months of development to get it to a working/reliable list.

- RSnake
Gotta love it. http://ha.ckers.org

Options: ReplyQuote
Re: Targeting Web Crawlers With PHP
Posted by: jungsonn
Date: January 10, 2007 04:27AM

Nice, would you care to share this list RSnake? could be nice to build an app around it.

edit:

But it got me wondering also; isn't there a signiature besides the IP used to detect a bot? I'dd like to learn more about that if there is a way how to. I read a few articles that do not scan every IP but base a sort of math around the first 2 octets to caculate if this could be a bot. Alas, I never bookmarked it, which is a pitty.
What I do know of personal experience is that if you do a IP lookup to get the host name, you mostly find that proxies have no hostname. This can be filtered upon and be analysed further, like for anti-spam methods.



Edited 1 time(s). Last edit at 01/10/2007 04:33AM by jungsonn.

Options: ReplyQuote
Re: Targeting Web Crawlers With PHP
Posted by: rsnake
Date: January 12, 2007 05:27PM

Heheh.... now you are seeing where I was going with this: http://www.secureseo.com/robot-mapping/ As you can see, I haven't updated it in several months (I have way way more records now) but I had more than 50k potential spider records already. I just needed someone to help out with the programming. Now I have more like 89k potential signatures, but I haven't put them online.

- RSnake
Gotta love it. http://ha.ckers.org

Options: ReplyQuote


Sorry, only registered users may post in this forum.