Web templates for web site design that saves YOU time and makes YOUR website look professional! Our website templates work as FrontPage templates, HTML Templates, Dreamweaver Templates...

Our website templates fit with most authoring programs including Microsoft Templates (Expression Web or FrontPage), Macromedia Dreamweaver Templates and more!

Web Template

Help Center Tutorials

Robots.txt File

<< Fonts Tutorial - Main Tutorial Page - CSS Tutorials >>

 

A robots.txt file is a simple txt file (created in notepad - to find this program click "Start - Run - Notepad in windows) that will help control and block certain files and/or directories from being crawled by the search engines.

For example, the "unused" folder we told you to create in the section "Removing & Adding Pages" section, you can use the robots.txt file to make sure it isn't crawled and indexed. You can also make sure certain pages are not indexed as well. For example if you create a .htm page you are not quite finished with, you can block it inside the robots.txt file until it is finished.

Here is what you need inside the .txt file (start with the blue section below)

User-agent: *
Disallow: /_private/
Disallow: /cgi-bin/
Disallow: /unused/

Continue to add lines with: "Disallow:" first. You can add as many as you need...

You can also add at the very bottom the following line:

Sitemap: http://www.yoursitehere.com/sitemap.htm

This will give you the ability to direct the search engines right to your sitemap.

Once you have finished putting together your robots.txt file, click "File - Save" and type robots.txt ... Make sure it is in the main folder (where all your .htm files are). Putting it inside any folders will not do you any good.

 

<< Fonts Tutorial - Main Tutorial Page - CSS Tutorials >>

 

 

Web Templates

 

Website Resources

 

Client Support

 

Company Information

 
By Category
Newest Templates
By Style
ALL Templates
Mini Template Categories
 
 
Resource Directory
Microsoft Expression Web
 
Template Disclaimer
Customizing Web Templates
Why Choose Our Templates?
FAQ's
 
 
Contact Us
Join Our Mailing List