Developing a new web site — or re-developing an old one, for that matter — typically involves consulting with the client to determine the site’s target audience and primary objective, creating an attractive and functional design, turning the design into properly coded, valid html, building out the pages of content, and writing the server-side programming to perform whatever dynamic features are needed. But there are several steps that are frequently overlooked before a site “goes live.”

A custom “404 not found” page should be created. At minimum, it should incorporate the site’s overall design and navigation links, and might also include a search form, links to the most popular sections of the site, and/or a way to contact the site owner or webmaster for assistance. And make sure that requests for non-existent pages actually get a “404 page not found” server response. If any other server response is returned — particularly a “200 OK” response — the site could easily become persona non grata in the search engines, among other problems.
A robots.txt file should be created to tell the search engine spiders what pages or parts of the site should not be spidered. Even if you want every page to be spidered, a robots.txt file should be placed in the document root of the site, so as to avoid filling up the site’s error logs with “not found” entries for a non-existent robots.txt. This makes it much easier to spot errors resulting from actual bad links when you examine the error logs.

A canonical URL redirect should be implemented to send all site traffic to your desired canonical URL — either www.example.com, or just example.com (without the www). Whichever you prefer, your should make sure that all traffic to the other form is redirected via a proper 301 redirect.

Test all forms and other interactive features. Submit every form. Attempt to submit forms without required information, or with invalid information. Try to break them. Try really really hard to break them. And make sure that whatever is supposed to be done after the form is submitted actually happens. If an e-mail is supposed to be sent to the site owner, test it, and make sure the owner gets that e-mail with all the appropriate information. If there’s a search engine, search for some things. If the site relies on cron jobs, set them up ahead of time, and make sure they’re running as scheduled and performing as expected.

Test all redirects and rewrites. If the site uses Apache’s mod_rewrite module to present search-engine-friendly URLs, test them all, and test the non-search-engine-friendly versions, to make sure that every bit of content can only be reached by one unique URL.

Check subdirectories for directory listings. Make sure that directory indexes are turned off, and/or for any subdirectory that doesn’t have an index page, plop one in there.

Test all internal links and all outgoing links. Make absolutely sure that every single link leads to the right place. We don’t need to stinkin’ dead links!

There’s simply no excuse for ever launching a site without having completed each of these steps.

URL (pronounced you-are-ell, or sometimes “earl” as in Duke of) stands for Uniform Resource Locator. It’s the technical name for the address of a particular web page. For example, the URL of this site’s home page is http://www.tropicalwebworks.org, and the URL of this page is http://www.tropicalwebworks.org/2007/01/18/infamous-canonical-url/.

It’s common that any particular web page may be reached at multiple URLs. If this site were not configured optimally, the home page might be reachable at both http://www.tropicalwebworks.org and http://tropicalwebworks.org (notice the missing “www.”). Normal people would logically think that this would be desirable: After all, you don’t want people to get a “server not found” error if they try to get to your site without including the www part.

But Google sees these as two completely separate URLs that just happen to contain exactly the same content. There are two problems with such a situation:

1. First, the “strength” of that page, and its ability to turn up in the search engine results, is diluted. Some of the page’s strength is allotted to one version, and some to the other, and neither “page” performs as well as it would if all the strength were concentrated in one page.
2. And second, Google attempts to filter out pages containing duplicate content, based on the reasonable logic that people don’t want to see multiple results in their searches for the exact same thing. Thus, since both of these “pages” contain the exact same content, one of them will suffer in searches due to the dupe content filter.

It’s a double whammy. It’s not that your site actually has duplicate content. No, we could possibly call this situation “virtual duplicate content.” But it’s all the same to Google: It’s duplicate content, period.

And if that’s not bad enough, many people link to their home page like this: http://www.example.com/index.html. Now Google sees yet another instance of duplicate content: http://www.example.com and http://www.example.com/index.html. So ultimately what Google sees is four “duplicate content” pages:

• http://www.example.com
• http://example.com
• http://www.example.com/index.html
• http://example.com/index.html

And all this before we’ve even gotten past the home page of your site!

It’s easy-peasy to configure the server to do what’s called a “301 permanent redirect” from the non-www version to the www version of your site. This technique, which is recommended by Google, tells Google that the two are indeed the same and keeps the poor Googlebot from deciding that you have duplicate content and splitting your page’s strength among more than one version. “301″ refers to the status code that’s returned by the web server to the browser (or the spider, in this case), and it says, in effect, “Hey, the correct, permanent URL for the page you’re requesting is actually over there. Don’t index it at this URL.”

It’s likewise easy-peasy to link to your home page without the “index.html” (or other directory index name, such as home.htm or default.asp). For index pages in subdirectories, you simply link to the directory: http://www.example.com/subdirectory/, again leaving out the actual filename index.html.

I apply an appropriate 301 permanent redirect to the www version of every web site I develop. It’s not something I charge extra for, or something that I tout to my clients as being anything special. It’s about a 20-second task to set up the 301 properly. And I never link to directory index pages by filename. I don’t know why some of the big companies aren’t aware of this issue, or, if they are aware, why they don’t care enough to do it properly. It raises the question, if they’re so ignorant, or uncaring, about a thing that is so simple to do right, in how many other areas are they incompetent?

These factors aren’t important so much for their ability to rank a site highly, as they are for avoiding problems that can harm a site’s ranking.

Using standard href links rather than javascript links, and avoiding canonical URL problems by implementing a 301 redirect from non-www to www, won’t actually help your site rank better for any given search. But using all javascript links and having canonical URL problems can harm your site’s ability to rank well.

These technological factors fall under the “Do no harm” umbrella. Don’t throw obstacles in the path of high rankings. Removing the obstacles won’t cause you to win that race — but it will make winning possible.