NYCPHP Meetup

[Chris Shiflett]

--- "DeWitt, Michael" <mjdewitt at alexcommgrp.com> wrote:
> For some of our sites, many of the pages require registration and login in
> order to view the page.  I would like to open up those pages to spiders in
> an effort to improve our web rankings.

You can either restrict access or not. A spider is basically an anonymous user,
so anything you do to let a spider view restricted resources removes the
restriction.

Sure, you can check the User-Agent header to see if it matches a known spider,
but your authentication is effectively reduced to someone sending this header,
and if you can find User-Agent strings for known spiders, so can an attacker.

Chris