I.T. Discussion Community!
-Collapse +Expand
Search Design Group:

-Collapse +Expand Design Store

Prestwood eMagazine

November Edition
Subscribe now! It's Free!
Enter your email:

   ► KBWebsite Scri...   Print This     
Next Random Article Next Random Tidbit || Change Topic Random From All

Random Tidbit

Part of the mo. lesson in the Jan 16 Issue of Prestwood eMag
Design Tip:
 A flashcard from our Design Flashcards Library
 A tip from our Design Tips Page
Don't Overlook Robots.txt

In the good old days, web spiders would crawl your sites once you registered them with a search engine.  Today, they are a lot more proactive, crawling sites when the domain names are registered.   For this reason, it is not optional during the development phase to add a robots.txt file to all projects that instructs robots not to crawl the site.

It's super easy.  Just create a text file in the website root directory named robots.txt. Put the following text in it.

User-agent: *
Disallow: /

That's it.  Now your temporary website is safe from most webcrawlers.  Note that all subdomains must have one of these in the root path.


Share a thought or comment...
Write a Comment...
Sign in...

If you are a member, Sign In. Or, you can Create a Free account now.

Anonymous Post (text-only, no HTML):

Enter your name and security key.

Your Name:
Security key = P1167A1
Enter key:
Tip Contributed By Bryan Valencia:

Bryan Valencia is the author of millions of lines of code and writes for PrestwoodBoards and Visual Studio Journey on-line magazines.

Visit Profile

 KB Article #101795 Counter
Since 1/6/2009

Follow PrestwoodBoards on: 

©1995-2019 PrestwoodBoards  [Security & Privacy]
Professional IT Services: Coding | Websites | Computer Tech