Search Engine Optimization 101: Sitemaps And Webmaster Tools

In this second article of my SEO 101 series, I will explain the process of creating a search engine specific XML sitemap. As well as, adding your site to search engines via Webmaster Tools. Finally, I cover a few of the basic Webmaster Tools concepts and features that will help you maintain your site's listings in search engines.

Setup A Search Engine Specific XML Sitemap

With a XML sitemap it is much easier for search engines (such as Google, Bing, Yahoo, and Ask.com) to crawl and index your website and blog. The sitemap gives the search engines a complete structured view of your site, indicating when the page / post was last updated and how often it changes, in addition to, what priority you have put on the page.

Search XML Sitemap Protocol

Create a sitemap following the XML sitemap protocol found at sitemaps.org. This is the format used by all the major search engines. Google also goes into great detail explaining sitemaps and their creation.

More References For Search Engine Crawling And Sitemaps

• Wikipedia's article on web crawler explains the basics of how search engine crawling works. A search engine crawls (finds) your pages by following the links in your site and links from other sites.
• It was estimated in 2005 that search engines only index 40-70% of the 11.5 billion publically available pages on the world wide web.
• Information on the Googlebot web crawler and how to make sure your site is crawlable can be found in the Google webmaster tools help documentation.
• You can use a free tool like XML-Sitemaps.com to create the sitemap for you. However this isn't going to be the best approach to optimize your sitemap.

XML Sitemap Plugins For Content Management Systems (CMS)

Many Content Management Systems (CMS) will have a sitemap plugin that you can install and use with minimal effort. They also properly adjust the modified date of each page to give search engines the best possible change of indexing your website.

• WordPress - Google XML Sitemaps; eventhough this has Google in the name it follows sitemaps.org protocol so it can be used for all the major search engines.
• CMS Made Simple - SiteMapMadeSimple
• Joomla has a bunch of XML Sitemap Extensions
• Drupal XML Sitemap Module

Submitting Your Sitemap To Search Engines Using Webmaster Tools

Submitting your sitemaps to the Webmaster Tools of each search engine will help. Once your sitemap is finished you will want to submit it to search engines. The best way to do this is by creating webmaster tools accounts with at least Bing and Google.

Please note that submitting an XML sitemap to any of these search engines is not a guarantee that your pages will be included in the search engine.

Webmaster Tools of Major Search Engines

You will have to signup with each of the search engines, and verify that you own your site, which can be done by uploading a file to your server, or adding a meta tag to your home page. Directions will be provided to you when you sign up.

• Bing Webmaster Tools -- [Bing setup help]
• Google Webmaster Tools -- [Google setup help]
• Yahoo Site Explorer even though Yahoo now uses Bing search, and the sitemap submission will be eliminated by 2012, Yahoo says their are some reasons to continue to use Site Explorer.
• Ask.com doesn't have a webmaster tool but the do have:
  • information for webmasters on how the Ask.com crawling and indexing works.
  • You can also submit your sitemap to Ask.com using this url: 
    http://submissions.ask.com/ping?sitemap=http%3A//www.yoursite.com/sitemap.xml

Basic Webmaster Tools Concepts

The main benefit of using the Webmaster tools is to analyze, monitor, and improve your site's Search Engine Optimization (SEO). Some of the things you can do:

• checking for duplicate content (like meta descriptions and title tags),
• submitting your XML sitemaps,
• getting information on which pages of your site have been indexed,
• what sites are linking back to your site (called backlinks or inbound links),
• tracking your keyword performance / search query keywords,
• and review the crawl errors found on your site (like 404 errors).

Webmaster Tools Help

There is a ton of documentation online about the specific uses of the webmaster tools. So instead of going into great deatail here are some references to help get you started.

• Google Webmaster Tools Help Reference
• Bing Webmaster Tools Help Reference

Robots.txt And Sitemap Auto-Discovery

The robots.txt file gives more information about your site to the search engine crawlers, and other web robots, about your site. Google Webmaster Tools also has a Generate robots.txt tool. Or if you want to build your own robots.txt file, can learn more about robots.txt at robotstxt.org. Robots.txt not only tells search engines what directories to include, it can also be used to tell search engines to skip directories to make their crawling more efficient.

If you are using a CMS you may already have a robots.txt file. So either create your robots.txt file or open the one that came with your CMS, and add the sitemap auto-discovery directive to your robots.txt file with one line of text right at the bottom of the file that says:

1. SITEMAP: http://www.yoursite.com/sitemap.xml

The above sitemap location should be the fully qualified sitemap URL.