Block Unwanted Visitors by IP Address or UserAgent in Apache using mod_rewrite

Use .htaccess rules to block unwanted bots, spiders and other UserAgents that don’t fetch, or that fetch and ignore robots.txt.

Blocking visitors by IP address filtering in .htaccess file:

# deny specific IP addresses, and allow all others
order allow, deny
deny from
deny from
deny from
allow from all

Block specific UserAgent using mod_rewrite

   # Block Google Images Bot from Indexing your Copyrighted Images
   # Hopefully someday Google will publish a "supported way" of
   # Disallowing the Google Image Bot when necessary, but until then...
   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Image
   RewriteRule ^(.*)$

The catch-22 with this method is that “sneaky” program developers can simply masquerade as “normal” visitors by using common web browser user agent strings. Reinforcing the fact that all three of these methods are USEFUL, but in no way a complete or secure solution even with the precise use of all three.

Also see:

  1. robots.txt
  2. Robots Meta Tag

Leave a Reply

Your email address will not be published. Required fields are marked *